Most voted "web-scraping" questions
It’s the process of extracting information from websites. It is typically used by third-party applications to extract information or interact with a website that does not expose an API.
Learn more…191 questions
Sort by count of
-
0
votes0
answers7
viewsHow to extract all comments from an instagram profile to an excel file?
Hello! I’m running a scraper to extract comments from open instagram profiles for an excel. However, the code I’m using extracts each comment to a notepad, separated into folders by post and…
-
0
votes0
answers19
viewsProblem extracting web page data with Beautiful Soup in python
I made a script in python to access the portal of records of the Inmetro to make a search among the existing certificates. In this case, my script accesses this link and takes all records from the…
-
-1
votes1
answer61
viewsSimple Html returns nothing?
Code: require_once("simple_html_dom/simple_html_dom.php"); html = new simple_html_dom(); $html->load_file('http://www.g1.globo.com/'); echo $html; Mistakes: Notice: file_get_contents(): send of…
-
-1
votes1
answer3786
viewsHow to press the button of a python website
I’m making a webscrapping of the club cipher site and picking up the most accessed songs of a genre. The code works normal but the way the page is made only appear the first 100 songs, to show the…
-
-1
votes1
answer88
viewsSimple_html_dom what is the difference between the two Urls?
Url2 works and can extract the data, Url1 does not. <?php include "simple_html_dom.php"; $CARDGALGO = file_get_html("$URLX"); echo $CARDGALGO; ?>
-
-1
votes1
answer2007
viewsProblem click python botan Selenium
I’m having trouble clicking a button, I tried it this way gives Elementnotvisibleexception: Message: element not interactable error pdf = driver.find_element_by_xpath('//*[@class="btn btn-default…
-
-1
votes1
answer258
viewsScraping data from a website with dynamic filtering
The search platform of the programs conceptualized in Capes has a dynamic filtering for the query itself. I would like to know how I collect data from an output using Python. Why, using only bs4…
-
-1
votes1
answer62
viewsRvest is returning the following error when reading a page: {xml_nodeset (0)}, how to resolve?
I would like to create a tool to make web scraping on the website of the Chamber of Deputies of Rio de Janeiro, but I am running into the problem of even being able to read the web page. Does anyone…
-
-1
votes1
answer37
viewsWebscript VBA array specifies
all right? I’m making a web scraping to update a spreadsheet of mine in vba. But the site I want to pull this information use td and tr and do not put class or id in the syntax. For example: I am…
-
-1
votes1
answer23
viewsHow can I get the value of a span tag with jsoup?
public class Main { public static void main(String[] args) throws IOException { // write your code here Document doc = (Document)…
-
-1
votes1
answer222
viewsI want to scrap a page, but I can’t get a text that has " " on it
Well, the title says it all, I want to get the price of a product from the site https://www.pontofrio.com.br/, like this: <span class="nm-price-value"…
-
-1
votes1
answer207
viewsHow to use scrapy on Asp.net pages
Well folks, my question is this:: I have to download the excel file of the product that contains the description "Maíz", Product type "Los Démas. En grano." and marketing "In bulk with 15 %…
-
-1
votes1
answer99
viewsModules of the Python3
I’m trying to make a Scrap on a college site, I usually do in Javascript with Casperjs and even then everything ok. But now we are trying to use Python for this and I’m having problems with this…
-
-1
votes1
answer101
viewsEmpty Dictionary Print - Webscrapping/python/xpath
Guys, I can’t understand why the result of this scrapp comes out an empty dictionary. Could help me understand what my mistake is? import requests from lxml import html quimicos = [] resp =…
-
-1
votes1
answer164
viewsHtml Agility Pack C#
I’d like to know how the // in the Html Agility Pack. Well, the doubt is the following, code: var t = htmlDoc.DocumentNode.SelectNodes("//table[@class='" + attr +…
-
-1
votes2
answers133
viewsHow I search properties within a td / tr using only js
Good afternoon guys, I need to search for data from within a site (scraping) that are within a table, with and . The information I need is within td’s. Follow the code: <table width="95%"…
javascript node.js javascript-events web-scraping scrapingasked 4 years, 12 months ago arthurgehrke 19 -
-1
votes1
answer61
viewsWeb Scraping iterator taking only elements with odd index
was making Webscraping from a sales page of cars and for some reason while iterating the data to collect the Mileage data round the iterator simply repeated the items, ie : 27000 27000 48000 48000…
-
-1
votes1
answer336
viewsSites with authentication - Web Scraping - Javascript
I am trying to automate a web data acquisition process using JS. In my case, I need to pull the information from the page https://sistema.justwebtelecom.com.br/adm.php. However, before going to this…
-
-1
votes1
answer90
viewsHow to stop a loop Each javascript
Hi, I’m developing a search engine for a bot rank on Twitch.tv follows the code below const cheerio = require('cheerio') const URL…
-
-1
votes2
answers999
viewsI can’t get elements from a page with Puppeteer
I’m creating an application to download an Instagram image through the URL and I’m using the package puppeteer to accomplish this task. Within the call of the method evaluate (method to run a JS…
-
-1
votes1
answer371
viewsPicking up texts within a column within a Python table
I always use variations of this code in other tables and it usually works, but this one I try anyway and it doesn’t work, what I’m doing wrong? url =…
-
-1
votes1
answer208
viewsDynamically load page contents
Basically I am trying to create a function that when receiving a name variable, looks for the name of the player on the site of the game and returns the information of that player if it exists in…
-
-1
votes1
answer118
viewsWeb Scraping with Pandas - How to treat values that are null in the collection and how to concatenate two columns in the final result?
I’m making a Web Scrap using Python and Pandas, on Windows. I am collecting the data from the page, generating a Dataframe in Pandas and then exporting to an Excel spreadsheet. I’m not using any…
-
-1
votes1
answer25
viewsRselenium - How to click the check-box and then the Submit button
I’m trying to make a web scraping of the Central Bank website. To do this, I need to select an item from the drop-down menu, then mark a check box and then click the Submit button. But when I try to…
-
-1
votes1
answer32
viewsHow to Scrap a Table and insert the data into the database?
Well, basically what I need is to do a microservice that extracts the prices from the soybean quotation from this table: https://www.canalrural.com.br/cotacao/soja/. And then insert the data into a…
-
-1
votes1
answer18
viewstake top element Selenium, python, scraping
<div class="cell-list-content-icon"> <span><i class="fa fa-briefcase"></i>EMPRESA XXXXXXXX</span> <span><i class="fas…
-
-2
votes1
answer411
viewsCrawler to scan websites
Talk Guys, all right? I would like to create a Crawler to scan day on some specific sites and bring me on a spreadsheet or something like the home stories of those sites. In case I’d like to do a…
-
-2
votes1
answer48
viewsWrite excel file within Scrapy project
I have a Spider that picks up the xlsx links, in Request I call the files and saved in: def save_file(self, response): f = open("teste.xls", "wb") f = write(response.body) f.close() But returns the…
-
-2
votes1
answer394
viewsWeb Scraping with python
Good evening. I want to make a simple algorithm to take data from a website (http://www.riooilgas.com.br/? _page=programming&_menu=programming). I’ve done some library code: from urllib.request…
-
-2
votes1
answer97
viewsGetting HTML attributes with python
I’m wanting to get the information from arial-label, href and title tag a down: <a aria-label="AS MAIS TOCADAS NO BAILE FUNK 2019 #1 - SET DE FUNK by Funk 24por48 10 months ago 39 minutes…
-
-2
votes1
answer46
viewsExcell VBA quickly corrseponde the other in another spreadsheet
need to run a macro depending on the selected cell. all the information is in the same column "E", but when it is for example for E4 I need the selected cell to correspond to another value in…
-
-2
votes1
answer421
viewsscraping google
Regards I am writing a script in python3 to perform news searches and tbm images... and created queries for Bing google using Apis mechanicalsoup and requests. the case is over with google, after a…
-
-2
votes1
answer65
viewsHow to capture data-tooltip using Selenium in Python
Hello, is the following my problem: I am making a web scraping in the following URL > https://www.maze.com.br/produto/tenis-nike-air-jordan-ma2-ultramarine-branco/4749595. I came across a slight…
-
-2
votes1
answer43
viewsWeb Scraping with Rvest - Problem with data scraping (links mainly)
I am trying to make a Web Scraping of the Web Site Of Science, but esstou facing problems with scraping links from the site. My intention is to scrape titles from the articles, links that direct to…
-
-2
votes1
answer62
viewsHow to read <br/> in HTML files and print as line breaks?
I made a web scraper using the modules BeautifulSoup and requests, that takes the definition and example of concepts in Urban Dictionary. This is code, using the word "reparation" as an example.…
-
-2
votes1
answer32
viewsBuilding a Web Scraping with Python
all right? Well, I’m a programming enthusiast, and I took a tutorial to make webscraping. The logic I even understood, however I am facing a problem when one of the data is missing on the site,…
web-scrapingasked 3 years, 4 months ago Guilherme Novais 3 -
-2
votes1
answer59
viewsHow to Scrap with Python to compare prices on a shopping site?
Comparing the price of the same product every minute and notifying me when the value drops 50%. I used a While >= but it is returning an error indicating that you cannot use >= between str and…
-
-2
votes0
answers19
viewsHow to make a scrapy post in python?
I am learning the lib Scrapy in Python and I am having difficulty to perform a request with the POST method for the url. I’m trying to use the following code: form={"letraLocalidade":"", "ufaux":"",…
-
-3
votes2
answers1313
viewsHow to extract a text from a page?
I am creating a web scraping program and would like to know if there is any way to extract the text from a site in this way: This is the html: <div class="s-sidebarwidget s-sidebarwidget__yellow…
-
-3
votes1
answer55
viewsOnly json content between []
I have a question about a code I made using scrapy to collect data and send to a json file. The problem is that the file formatting is not as it usually is, so I found it strange, I’m in doubt if…
-
-3
votes0
answers61
viewsHow to make web scraping with Selenium on all pages of a website?
I’d like to do web scraping with Selenium on every Free Market offering page, but so far I’ve only been able to do it on the first one. I use Pandas to store the data in a dataframe. How do I do it…