I can’t get elements from a page with Puppeteer

Question

I’m creating an application to download an Instagram image through the URL and I’m using the package puppeteer to accomplish this task.

Within the call of the method evaluate (method to run a JS code on the page) I try to get and return an image element.

The problem is that whenever I try to get the image, I end up getting null as a result. See my code below:

const puppeteer = require("puppeteer");
const url = "https://www.instagram.com/p/<image_code>/";

async function getImageFrom(url) {

    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.goto(url);

    const image = await page.evaluate(function() {
        return document.querySelector(".KL4Bh > img");
    });

    await browser.close();
    return image;
}

getImageFrom(url).then((image) => {
    console.log(image);
});

What am I doing wrong? I’ve tried using querySelectorAll searching only for elements with the tag <img> but it also returns me only an empty array.

Edit: I don’t know what happened, but the method is returning undefined now.

Answer 1

You probably DON’T is waiting for the load to complete or it may be that the way instagram loads is dynamic and even after the load, each site makes a way.

Change this:

const image = await page.evaluate(function() {
    return document.querySelector(".KL4Bh > img");
});

For:

page.waitForSelector(selector[, options])

To wait for the "exist" element that is already a function native (evaluete would be JS within the generated webview, which is unnecessary rework), example:

page.waitForSelector('.KL4Bh > img');

Or instead of trading seven of the property waitUntil in options

page.goto(url[, options])

And in it apply the value networkidle2 (which would be like load and also no more than 2 network connections for at least 500ms), example:

await page.goto('https://site.com', { waitUntil: 'load' });

Answer 2

The method evaluate is returning the value null because maybe the element doesn’t exist on the page. Either because it hasn’t been loaded yet or because it doesn’t really exist on the current page even though it has loaded all the JS code.

The other reply says to set the value "load" for the option waitUntil, but this is redundant since according to documentation of this API, by default this option will be "load".

Still this is not the main problem of your code. Even if the element has loaded on the page, the method evaluate must return undefined if all goes well. That’s because you’re trying to return a non-serializable object.

If your goal is just to get the attribute src or another of that element that returns a string or another serializable value, do it within the method and return it as in the code below:

const image = await page.evaluate(function() {
    return document.querySelector(".KL4Bh > img").src;
});

If you need the whole element and not just an attribute or simple value, use the method waitForSelector as shown in the other reply. What this method does is wait and return an element of the page with the selector you set. See the code below:

const image_element = await page.waitForSelector('.KL4Bh > img');

This method will return an object from ElementHandle. To get the attributes of this element, use the method getProperty(attr) which returns an object of JSHandle.

After having obtained this object, use the method jsonValue() to obtain the attribute value.

const image_element = await page.waitForSelector('.KL4Bh > img');
const image_property = await image_element.getProperty("src");

const src = await image_property.jsonValue();

Remember that all these actions should be done with the browser open. Therefore, you cannot close the browser before obtaining the image property or you will receive an error.