A fairly intuitive library to control a browser by Python (from accessing urls to getting page-specific elements and clicking buttons) is the Selenium.
To install it for Python 2.*:
pip install selenium
And for Python 3.*:
pip3 install selenium
After installation for Python 3.*, you can do as follows:
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Firefox()
driver.get("https://www.cifraclub.com.br/mais-acessadas/")
btn_more = driver.find_element(By.ID, 'js-top_more')
btn_more.click()
Clicking the button in question will display 20 new songs each click, and keeps the same ID (which can be quite practical for you), so analyze how many songs you plan to load.
Another tip I can give you is to use the Webdriverwait, that makes the driver expect the appearance of an element based on a delay time and expected_conditions, that will offer conditions for Webdriverwait, which may be fundamental between a click and another on the button.
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://www.cifraclub.com.br/mais-acessadas/")
while(sua_condicao):
btn_more = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.ID, 'js-top_more')))
btn_more.click()
Are you using Selenium?
– Pedro von Hertwig Batista
I’m just using the beautifulsoup
– Dhonrian
With
beautifulsoup
it doesn’t make much sense to think about clicking buttons. You have two alternatives: use Selenium (control a real browser with code) or analyze and replicate site requests using your browser’s developer tools.– Pedro von Hertwig Batista
Tip: Try contacting the site owner to see if there isn’t a public API you can use. Usually web-Crapping is something unauthorized in the terms of use of most websites.
– Pagotti