Web Scrapping Nodejs - I cannot use variable as parameter

Question

I’m using the library Nightmare, for Web Scrapping, function works normally when I pass a string as parameter '.seletor', the problem occurs when I store this value in a variable and step as parameter, follows code below (I will use a product from Amazon as an example):

JS:

const nightmare = require('nightmare')()

async function checkPrice () {
      const url = 'https://www.amazon.com.br/250GB-SAMSUNG-970-EVO-PCIe/dp/B07CGGNX7S/'

      const priceNow = await nightmare.goto(url)
        .wait('#priceblock_ourprice')
        .evaluate(() => document.querySelector('#priceblock_ourprice').innerText)
        .end()

      console.log(priceNow) // R$664,90
    }

checkPrice()

This way the product price is returned, without any kind of error...

Follow example with the error:

JS:

const nightmare = require('nightmare')()

async function checkPrice () {
      const url = 'https://www.amazon.com.br/250GB-SAMSUNG-970-EVO-PCIe/dp/B07CGGNX7S/'
      const priceSelector = '#priceblock_ourprice'

      const priceNow = await nightmare.goto(url)
        .wait(priceSelector)
        .evaluate(() => document.querySelector(priceSelector).innerText)
        .end()

      console.log(priceNow) // Mensagem de erro
    }

checkPrice() 

Error message:

Answer 1

After a little research, I found this document. In accordance with I pondered on the comments, the method evaluate really does a certain magic behind the scenes, which causes all this confusion.

What happens is that as this library uses Electron behind the scenes (somehow I didn’t try to understand). Therefore, it transforms the callback function that you pass to as an argument for the method in question so that it can be executed in the context of Electron. That’s why even though it seems to be in the scope, variavies like the priceSelector are not, since the function will be running only in the Electron context. This is also why passing literal values does not cause any kind of error: they are also placed in the function "stringificada".

Behold on Github the moment the callback function is converted to string.

So you have two alternatives to solve the problem:

Using the last arguments of evaluate to provide parameters for the callback that will be evaluated

As stated in documentation of evaluate and in the document that I mentioned earlier, you can pass other arguments to the evaluate, which will be passed to your callback. Something like this:

// ...
.evaluate(
  (selector) => document.querySelector(selector).innerText,
  priceSelector
)
// ...

Note that now callback function is receiving selector as parameter. Parameter that we pass as argument to evaluate after the callback. ^[4]

Pass literal values

If you find it simpler, you can also pass the form values literal. Basically, if you prefer this option, you will only be restricted to using any kind of external reference to the callback scope.

---

Anyway, it’s up to you to choose which option you prefer. :-)