Scrapy queueQueue and mysql store

Question

I’ve grouped 2 questions because I think they’re related.

I made a script test, where saved in the database the links saved with your data.

This is a bad practice? (High priority)

Do I need to accomplish something else so I don’t import duplicates? In my pipeline has a simple check looking for link=%s, would it be better if I use md5 (link)? Faster query?

I can use the -s JOBDIR=crawls/somespider-1 to pause and return Crawler, but would like to know how to accomplish this by list of links to be processed in Mysql. (Low priority)

I need to add new items to my list of start_urls, or dynamically. I must create Request with callback parse_category? Is there any way I can add self.queue or self.start_url and add new url’s to be processed? (High priority)

Answer 1

This is a bad practice?

No, considering the responsibility for handling the Mysql connection is also being done using good Python practices.

I can use the -s JOBDIR=crawls/somespider-1 to pause and return the Crawler, but would like to know how to accomplish this by list of links to be processed in Mysql.

First you select Mysql records. Then you can use some repeat loop to go calling requests and their respective callbacks:

for registro in registros:
    resultado = Request(url=self.registro['url'], callback=self.meu_callback)
    # Aqui você faz operações adicionais com resultado, se precisar.

I must create Request with callback parse_category?

It’s the right thing to do.

Is there any way I can add self.queue or self.start_url and add new url’s to be processed?

self.queue you will bring from your database. self.start_url will be the column of each record brought.