Posts by Renan Gomes • 11 points
3 posts
-
0
votes2
answers126
viewsA: How to calculate an optimal value for Scrapyd’s CONCURENT_REQUESTS variable?
The number can vary a lot from server to server. And, if you are creating a generic server to index multiple sites, it gets even harder. Normally I keep increasing until I can make my CPU close to…
-
0
votes2
answers113
viewsA: How to protect my Scrapyd server from unauthorized calls?
You can edit the scrapyd settings and put the following configuration in the ~/.scrapyd.conf file: bind_address = 127.0.0.1 This will make the server only usable by processes running on itself. If…
-
1
votes1
answer37
viewsA: Scrapy 1.0 - Log Settings
To change the values of the statistics generated during the Spider execution, you must use the "Stats Collection" interface. Handbook: http://doc.scrapy.org/en/latest/topics/stats.html Examples:…