Code improvement

Question

Hello. I have this code:

import os
from selenium import webdriver  
import time



def visita(link):
    try:    
        options = webdriver.ChromeOptions()
        options.add_argument('--headless')
        driver = webdriver.Chrome(options=options)
        driver.get(link)
        driver.execute_script('window.scrollTo(0,document.body.scrollHeight);')
        time.sleep(5)
    except Exception as e:
        print(e)
    driver.close()


links = ['https://link.com/post/6074/','https://link.com/post/6288/']
while True:
        for link in links:
                visita(link)

Code summary:
1- Open links using chromedriver
2- Run a javascript on the page to scroll to the bottom of the page, wait 5 seconds and close with driver.close()

Problem: Code execution is quite slow.

Is there any other module/library that can do the same more effectively ?

Answer 1

Selenium is already slow by nature because it needs to upload its application, some cases with database, open Chrome, and browse, you even added for it to stay 5 seconds waiting that will slow this whole process 5 seconds.

To optimize you can use Phantomjs instead of Selenium, Phantomjs emulates a browser, and review the need to wait 5 seconds to close the driver, probably less than a second should be enough for you.

Answer 2

If you are scraping on sites that do not require user authentication, that is, you do not need to be logged in to have access to these links, you can simply use the Beautiful Soup, saves you a lot of running time than using Selenium.

Now as mentioned by Roberto, because it is Selenium, the existence of a physical browser open on the machine is already consuming a lot of time, what Voce can do as he said and emulate a browser without the need to open a webdriver. You can do this both with the Mechanize for python2 as with the Mechanicalsoup for python3.

Another tip I give, and because it is multiple links, you can use threads and make several scrapings at once, speeding up the whole process. Of course the page loading time still remains the same thing, or if not worse depending on your internet link.