How to press the button of a python website

Question

I’m making a webscrapping of the club cipher site and picking up the most accessed songs of a genre.

The code works normal but the way the page is made only appear the first 100 songs, to show the rest I have to physically click the "Show More" button of the site. Pressing the button doesn’t change anything in the url I can use in the program to access more pages.

<button id="js-top_more" class="btn btn-full g-sb">mostrar mais</button>

And after that there’s nothing else related to the button.

Is there any way around this problem?

Answer 1

A fairly intuitive library to control a browser by Python (from accessing urls to getting page-specific elements and clicking buttons) is the Selenium.

To install it for Python 2.*:

pip install selenium

And for Python 3.*:

pip3 install selenium

After installation for Python 3.*, you can do as follows:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.cifraclub.com.br/mais-acessadas/")

btn_more = driver.find_element(By.ID, 'js-top_more')
btn_more.click()

Clicking the button in question will display 20 new songs each click, and keeps the same ID (which can be quite practical for you), so analyze how many songs you plan to load.

Another tip I can give you is to use the Webdriverwait, that makes the driver expect the appearance of an element based on a delay time and expected_conditions, that will offer conditions for Webdriverwait, which may be fundamental between a click and another on the button.

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.cifraclub.com.br/mais-acessadas/")

while(sua_condicao):
    btn_more = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.ID, 'js-top_more')))
    btn_more.click()