Scraping data from a website with dynamic filtering

Asked

Viewed 258 times

-1

The search platform of the programs conceptualized in Capes has a dynamic filtering for the query itself. I would like to know how I collect data from an output using Python. Why, using only bs4 libs and requests I can not get output of this data (I believe precisely because it is a dynamic query). On behalf of JS and Ajax.

Consultation site CAPES programs

In Python, how could you scrape the data of this query with a specific filter ?

EDIT: Well, apparently, I’ve been able to learn a little bit more about the Selenium lib, and I’ve come a long way. My code until then:

from selenium import webdriver
from selenium.webdriver.support.ui import Select

chrome_path=r"C:\Users\USERNAME\Desktop\chromedriver_win32\chromedriver.exe"
driver=webdriver.Chrome(chrome_path)
driver.get("https://sucupira.capes.gov.br/sucupira/public/consultas/coleta/programa/listaPrograma.jsf")
driver.find_element_by_id("""form:checkAreaAv""").click()
driver.find_element_by_xpath("""//*[@id="form:autocompleteAreaAv:input"]""").send_keys("CIÊNCIA DA COMPUTAÇÃO")
Select(driver.find_element_by_xpath("""//*[@id="form:autocompleteAreaAv:listbox"]""")).select_by_value(2)
f.find_element_by_xpath("""//*[@id="form:consultar"]""").click()
results=f.find_elements_by_class_name("resultados")

However, I’m having a hard time selecting an item from a drop-down menu in the text itself Area of Evaluation.

ajax python python-3.x web-scraping scraping

  • Your question is too Enerica, what have you tried? what problems have you found? or would this question be totally outside the context of this "board"?

  • I have tried using lib request + Beautiful Soup. However, it does not indicate any result, even indicating the exact HTML path.

  • Show me the code you tried.

  • Well, I followed my friend Denis' tip down there, and I was able to learn how to handle the Lenium webdriver and capture the data. Anyway, I still could not because in the middle of the way I face a text with a drop-down menu of options to filter the result, but I think I’m on the right path now

  • I repeat... show me the code you tried and what was the difficulty, if you don’t want to be adjuncted, you want them to solve your problem. Sad.

  • I’ve edited the question now, with my code so far.

  • My question would be when I use the select function of Selenium, it ends up not selecting by listbox value.

Show 2 more comments

1 answer

0


Maybe you could use the Selenium to emulate the use and capture the data. If the amount of data to be captured is not too large, surely Selenium can help you.

  • I used Selenium, however, my code still remains something I’m having trouble solving, I edited the question for the demonstration of it.

  • Do the following: 1. Insert something into the field until a dropdown result appears. 2. Right-click the result and choose to inspect. 3. You will have field information. Then study the best way to capture it with Selenium, based on the field type.

Browser other questions tagged ajax python python-3.x web-scraping scraping

You are not signed in. Login or sign up in order to post.