Scrapy 1.0 - Log Settings

Asked

Viewed 37 times

2

I need to know how to change the highlighted fields, because when I run my program with Scrapy in version 1.0 it prints the result in these highlighted quantities.

inserir a descrição da imagem aqui

I wanted to know how to change them, whether it is through a script or Scrapy settings or even Spiders.

python debug scrapy

  • Hello, Caio! Did you have any problem in putting into practice Renan’s answer? The Stats are like a global object for each job, so just get a reference to them and can change from any type of component (Spider, pipeline, middleware, etc).

1 answer

1

To change the values of the statistics generated during the Spider execution, you must use the "Stats Collection" interface. Handbook: http://doc.scrapy.org/en/latest/topics/stats.html

Examples:

  1. Change a value:

    stats.set_value('hostname', socket.gethostname())

  2. Increase:

    stats.inc_value('pages_crawled')

The 'Stats' object is a field of the 'Crawler' object'.

crawler.stats.get_value('start_time')

Browser other questions tagged python debug scrapy

You are not signed in. Login or sign up in order to post.