How to press the button of a python website

Asked

Viewed 3,786 times

-1

I’m making a webscrapping of the club cipher site and picking up the most accessed songs of a genre.

The code works normal but the way the page is made only appear the first 100 songs, to show the rest I have to physically click the "Show More" button of the site. Pressing the button doesn’t change anything in the url I can use in the program to access more pages.

<button id="js-top_more" class="btn btn-full g-sb">mostrar mais</button>

And after that there’s nothing else related to the button.

Is there any way around this problem?

  • Are you using Selenium?

  • I’m just using the beautifulsoup

  • With beautifulsoup it doesn’t make much sense to think about clicking buttons. You have two alternatives: use Selenium (control a real browser with code) or analyze and replicate site requests using your browser’s developer tools.

  • Tip: Try contacting the site owner to see if there isn’t a public API you can use. Usually web-Crapping is something unauthorized in the terms of use of most websites.

1 answer

1


A fairly intuitive library to control a browser by Python (from accessing urls to getting page-specific elements and clicking buttons) is the Selenium.

To install it for Python 2.*:

pip install selenium

And for Python 3.*:

pip3 install selenium

After installation for Python 3.*, you can do as follows:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.cifraclub.com.br/mais-acessadas/")

btn_more = driver.find_element(By.ID, 'js-top_more')
btn_more.click()

Clicking the button in question will display 20 new songs each click, and keeps the same ID (which can be quite practical for you), so analyze how many songs you plan to load.

Another tip I can give you is to use the Webdriverwait, that makes the driver expect the appearance of an element based on a delay time and expected_conditions, that will offer conditions for Webdriverwait, which may be fundamental between a click and another on the button.

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.cifraclub.com.br/mais-acessadas/")

while(sua_condicao):
    btn_more = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.ID, 'js-top_more')))
    btn_more.click()

Browser other questions tagged

You are not signed in. Login or sign up in order to post.