1
Using Htmlagilitypack, I use the following code to get the HTML of a website:
var url = "teste.com";
var web = new HtmlWeb();
var doc = web.Load(url);
However the data (a table) I want to extract from the site, appears only in inspect browser element and not in source code... How can I get all the contents of the inspect element?
Have you checked if it is not in an iframe of the page? I think the ideal would be to have more information.
– Lucas Miranda
Here’s the source code my code obtains https://pastebin.com/j7rix872 (I think there’s javascript involved, does that influence?)
– Alan Assis
Yes, it influences, some of the contents can be inserted later with javascript. Ideally, you should map the requests and see if any of them after the page request are picking up this information. I will pass as an example the site checkerproxy.net that seems to be similar or even what you are using, when looking at the network tab here of Chrome I saw that it makes a request that returns the page and then another that returns a json with the data to fill in next.
– Lucas Miranda
Is there any way to get this data? I’m trying and I haven’t made any progress
– Alan Assis
Yes, of course, but for that you would need to understand the way you’re trying to pick up, if it’s for example via httprequest it would be a matter of changing the endpoint you’re hitting that the answer would be what you want.
– Lucas Miranda
I was clicking here and there on the network tab, since I didn’t know her before, and I ended up finding the site https://checkerproxy.net/api/archive/2018-09-06
– Alan Assis
Simply put, this link contains what I need
– Alan Assis
Thanks, that solved my problem...
– Alan Assis