How to read the html of a page that only loads when opening in GOLANG browser

Asked

Viewed 61 times

-2

I’m trying to read the HTML from the following page (https://www.lojacorpoperfeito.com.br/search/? search=supplements&index=0/), but the problem is that its HTML is only loaded when it is opened in the browser, when entering it and pressing Ctrl + U you will notice that there is only the javascript that loads the products, but in the HTML it stays as it should be to be read.

I am currently opening the pages with "goquery".

golang

  • Shows what you’ve tried, can help...

  • tried the usual, searching for page tags to try reading the href content, but the open page in the browser is different from the one opened by GO, so I can’t find any tags I’m looking for

1 answer

1


You already answer your question yourself. You say "by entering it and pressing Ctrl+U you will notice that there is only javascript that loads the products".

Golang by default has no Javascript engine and in a general context there is no point in having it. You are probably getting the contents of the page using the http, it will only get the content of that page, ie the same HTML you are seeing in CTRL+U. If it contains any <img>, <script> (...) will be insignificant, Golang will not upload this content unless you use some library that does this (some "golang browser" or integrate it with some other webdriver...).

The most practical solution is to see where the content is obtained. If the element is dynamically created it comes from somewhere, then just find this source.

The most practical way is to go in the Console (F12 or CTRL + SHIFT + I) and then in "Network" and select "XHR", in this case you will see that there is a request for:

https://busca.saudifitness.com.br/externalapp/?busca=SeuTermoBuscado&full=true&idapp=1

It returns a JSON with the product, it is the content that JS uses to display the elements on the page. Ready. Instead of ordering the other page (which contains the JS), simply request the page that already has the information...

  • was exactly what I did friend, thank you for reminding me of the question, I will give as correct even having discovered before because it is exactly that, thank you

Browser other questions tagged golang

You are not signed in. Login or sign up in order to post.