Most voted "web-crawler" questions
The Web Crawler (also known as Web Spider) is a computer program that navigates the World Wide Web in a methodical and automated manner or in an orderly manner. Other terms for web crawlers are Ants, automatic indexers, bots, spiders/Spiders, web robots, or - especially in the FOAF community - Web scutters.
Learn more…66 questions
Sort by count of
-
0
votes1
answer104
viewsMechanize with Nokogiri: trying to get information on Ivs
Hello! I am assembling a Crawler for product information, for this I am using mechanize and consequently Nokogiri, I have a URL (http://www.megamamute.com.br/brother%205652) that returns only one…
-
0
votes1
answer62
viewsFind out which Encode was used in a form
I’m making a request for the site: http://sistemas.cvm.gov.br/? PAS Before making the request, they make a Match of the filters. For example, when I pass Unfair practices he shrunk to…
ruby-on-rails ruby character-encoding web-crawler encodeasked 4 years, 9 months ago Willian Lucio 1 -
0
votes3
answers868
viewsCatch string inside <a> tag without attributes
I’m using gift in PHP to get the link of a tag , where through "getattribute" I can get that link by the href attribute. Script by Crawler: <?php //carregamento da url $dom = new DOMDocument();…
-
0
votes1
answer177
viewsHelp Chrome Puppeteer Paging
I’m doing a Webcrawler, in the following stream Access a search URL Collecting the information Execute pagination (here is the error) Click to the next page, reload it (then I tried to force a…
-
0
votes0
answers45
viewsRobots continues to index deleted pages
I work in a company that owns a store Magento 1.9 and in order to optimize the search organizes we are trying to delete hundreds of pages indexed by google, however, I am facing two difficulties.…
-
0
votes1
answer733
viewsScrapy for login
I took this code from the internet and changed it a little, to log in to the cpfl site, but when I use the command scrapt crawl myproject nothing happens and the command scrapy runspider items.py of…
-
0
votes1
answer485
viewsHow can I use Scrapy in Anaconda
Hi, I’m having trouble creating a project with Scrapy. I’m studying data science in college and I have to use Scrapy. I’m using Anaconda. First through the Spider IDE (Anaconda Navigator), now I’m…
-
0
votes1
answer161
viewsCrawler - how to access several pages
I created a code on Ode to search for the version of the system and the name of the municipality of a portal, however I am not able to make it search for the information of another municipality only…
-
0
votes1
answer59
viewsHow to name each row of a url list
I can name every line on the url list, to return the nickname I gave her? Like that was the result: Prefeitura Municipal de Bocaiúva do Sul | PRONIM TB 518.01.07-013 | Prefeitura Municipal de…
-
0
votes1
answer397
viewsTweeter
I am using the API provided by Tweeter next to python to fetch certain tweets. The problem is that I wish view the tweets received by the person and not the tweets sent by them however, I am not…
-
0
votes0
answers118
viewsProblem collecting links from a website
Dear friends, good morning! I am writing a program in Python to collect the links of a website. The part of the code that collects the links is: links = driver.find_elements_by_xpath('//*[@href]')…
-
0
votes1
answer112
viewsData Crawling in Python
Good afternoon, you guys. I decided to start my studies with the Python Crawler technique. I built the following script using lib Selenium : # Importando selenium para realizar o crawling from…
-
0
votes1
answer242
viewsCatch second row of a table
I’m developing an application using Node.js and puppeter to take data from a web page, the problem I have and that I’m not getting the columns I want inside the selected row, this is the part of the…
-
0
votes1
answer28
viewsProblems with INSERT INTO in Crawler in PHP - Error passing variable as parameter
<?php require_once "./vendor/autoload.php"; use Goutte\Client; $servername = "localhost"; $database = "*******"; $username = "root"; $password = "********"; $conn = new mysqli($servername,…
-
-1
votes1
answer237
viewsCrawler
Currently I develop Crawler in php using guzzle, and lately I’ve been looking if it would be possible to use javascript to create Crawler, using the user’s ip and such, you know if it is possible?…
-
-2
votes1
answer411
viewsCrawler to scan websites
Talk Guys, all right? I would like to create a Crawler to scan day on some specific sites and bring me on a spreadsheet or something like the home stories of those sites. In case I’d like to do a…