Create a new function with scrapy

Question

I’m starting to learn scrapy and created the following function:

import scrapy


class ModelSpider(scrapy.Spider):
    name = "model"
    start_urls = [
        'http://www.icarros.com/'
    ]

def parse(self, response):
    with open('brands.csv', 'r') as all_brands:
        for brand in all_brands:
            brand = brand.replace("\n", "")
            url = 'http://www.icarros.com/'+brand
            yield scrapy.Request(url, self.success_connect)

def success_connect(self, response):
    self.log('Entrei')

But the following error appears:

AttributeError: 'ModelSpider' object has no attribute 'success_connect'

Answer 1

There is a problem in the indentation of your file. The two functions are outside the class (as Python does not have { }, is the indentation that defines the code blocks). The file carros_spider.py working gets like this:

import scrapy


class ModelSpider(scrapy.Spider):
    name = "model"
    start_urls = [
        'http://www.icarros.com/'
    ]

    def parse(self, response):
        for brand in ['ford', 'toyota']:
            brand = brand.replace("\n", "")
            url = 'http://www.icarros.com/'+brand
            yield scrapy.Request(url, self.success_connect)

    def success_connect(self, response):
        self.log('Entrei')

To execute:

scrapy runspider carros_spider.py

See the output of this command.