Django application (NGINX and WSGI) consuming only one core, reaching 100% and locking

Question

I have a problem that I have deployed with NGINX and UWSGI on a debian 9 with 24 cores, the application is simply running correctly, but when I upload an excel file the application is using only one server core and with it reaching 100%, with this the application takes time to process the file giving the times time out.

py views.

def create(self, request, *args, **kwargs):
    phone_numbers = []
    count_error = 0
    phone_invalid = 0
    count_repeat = 0
    count_success = 0
    nodes_success = []
    nodes_error = []
    black_list_logs_saved = []
    black_list_saved = []
    vicidial_dnc_logs = []
    phones_unique_saved = []
    logs_saved = []

    try:
        date_init = datetime.datetime.today()
        file = request.data['file']

        SaveFileExcel.objects.create(
            user=request.data['user'],
            action=request.data['action'],
            file_path=file
        )

        try:
            read_file = pd.read_excel(file)
        except Exception as e:
            _queryset = SaveFileExcel.objects.filter(action__icontains='add')
            _serializer = SaveFileExcelSerializer(_queryset, many=True)
            read_file = pd.read_csv(os.path.join(BASE_DIR, _serializer.data[-1]['file_path']))

        phones_in_file, phones_invalid, amount_phones = utils.treating_numbers_in_file(read_file,
                                                                                       BlackListSerializer(
                                                                                           BlackList.objects.all(),
                                                                                           many=True))
        phones_uniques, phones_repeat = utils.treating_numbers_replay(phones_in_file)

        for phone in phones_invalid:
            vicidial_dnc_logs.append({'phone_number': phone, 'campaign_id': '-SYSINT-',
                                       'action': 'add', 'action_date': str(datetime.datetime.today()),
                                       'user': request.data['user']})
            black_list_logs_saved.append({'phone_number': phone, 'user': request.data['user'], 'action': 'Invalid'})
            logs_saved.append({'action': 'invalid', 'user': request.data['user'], 'status': 'Alert',
                               'message': 'Phone number {0} invalido'.format(phone), 'method': 'blacklistlogs'})
            with open(_file, 'a') as arq:
                arq.write('\n[{0}] - Phone number {1} invalido.'.format(
                    datetime.datetime.today(), phone))

        for phone in phones_uniques:
            phones_unique_saved.append({'phone_number': phone})
            vicidial_dnc_logs.append({'phone_number': phone, 'campaign_id': '-SYSINT-',
                                      'action': request.data['action'], 'action_date': str(datetime.datetime.today()),
                                      'user': request.data['user']})
            black_list_logs_saved.append({'phone_number': phone, 'user': str(request.data['user']), 'action': request.data['action']})
            logs_saved.append({'action': 'Add', 'user': request.data['user'], 'status': 'Success',
                               'message': 'Phone number {0} adicionado'.format(phone), 'method': 'blacklistlogs'})
            with open(_file, 'a') as arq:
                arq.write('\n[{0}] - Phone number {1} inserido.'.format(
                    datetime.datetime.today(), phone))
        with open(_file, 'a') as arq:
            arq.write('\n[{0}] - Foi encontrado {1} números repetidos na lista inserida'.format(
                datetime.datetime.today(), phones_repeat))

        logs_saved.append({'action': 'insert', 'user': str(request.data['user']), 'status': 'Success',
                           'message': 'Foi inserido {0} números no banco.'.format(phones_uniques.__len__()), 'method': 'blacklistlogs'})
        with open(_file, 'a') as arq:
            arq.write('\n[{0}] - Foi inserido {1} números no banco.'.format(
                datetime.datetime.today(), phones_uniques.__len__()))
        date_end = datetime.datetime.today() - date_init
        with open(_file, 'a') as arq:
            arq.write('[{0}] - Processamento de dados em {1} tempo.'.format(datetime.datetime.today(),
                                                                                      date_end))
        print('processando dados em : {0}'.format(date_end))
    except Exception as ex:
        request_data = []
        try:
            if '.txt' in str(file.name):
                SaveFileExcel.objects.create(
                    user=request.data['user'],
                    action=request.data['action'],
                    file_path=file
                )

                _queryset = SaveFileExcel.objects.all()
                _serializer = SaveFileExcelSerializer(_queryset, many=True)
                file_path = os.path.join(BASE_DIR, _serializer.data[-1]['file_path'])

                with open(file_path, 'r') as arq:
                    list_data = arq.read().split('\n')
                    request_data = list_data[1:]
                request_data = utils.treating_txt_request(request_data, str(request.data['user']), str(request.data['action']))
        except UnboundLocalError as unex:
            request_data = request.data

        if type(request_data) is list and request_data.__len__() > 0:
            user = request_data[0]['user']
            for black in request_data:
                if {'phone_number': str(black['phone_number'])} not in phone_numbers \
                        and BlackList.objects.filter(phone_number=black['phone_number']).exists() is False:
                    phone_numbers.append({'phone_number': str(black['phone_number'])})
                else:
                    count_repeat += 1
                if nodes_server['nodes'].__len__() > 0:
                    for node in nodes_server['nodes']:
                        if node == 'teste':
                            try:
                                vicidial_dnc_logs.append({'phone_number': black['phone_number'], 'campaign_id': '-SYSINT-',
                                                          'action': black['action'], 'action_date': datetime.datetime.utcnow(),
                                                          'user': user})
                                if node not in nodes_success:
                                    nodes_success.append(node)
                                with open(_file, 'a') as arq:
                                    arq.write('[{0}] - Success - Phone number {1} gravado no node {2}\n'.format(
                                        datetime.datetime.today(), black['phone_number'], node))
                            except Exception as e:
                                if node not in nodes_error:
                                    nodes_error.append(node)
                                count_error += 1
                                with open(_file, 'a') as arq:
                                    arq.write('[{0}] - Error - Phone number {1} não foi gravado no node {2} - Error: {3}\n'.format(
                                        datetime.datetime.today(), black['phone_number'], node, e))
                if 9 < str(black['phone_number']).__len__() < 12:
                    black_list_saved.append({'phone_number': black['phone_number']})
                    black_list_logs_saved.append(black)
                    with open(_file, 'a') as arq:
                        arq.write('[{0}] - Phone number: {1} inserido\n'.format(
                            datetime.datetime.today(), black['phone_number']))
                else:
                    with open(_file, 'a') as arq:
                        arq.write('[{0}] - Numero {1} incorreto.\n'.format(datetime.datetime.today(), black['phone_number']))
                    utils.save_logs_blacklist('insert', user, 'Invalid',
                                              'Numero {0} invalido, nao foi inserido.'.format(black['phone_number']),
                                              black['phone_number'])
                    phone_invalid += 1

            if nodes_server['nodes_error'].__len__() > 0:
                for node in nodes_server['nodes_error']:
                    logs_saved.append({'method': 'blacklistlogs', 'action': 'insert', 'user': user, 'status': 'Error',
                                       'message': 'Não foi possível inserir dados no node: {0}.'.format(node)})
                    count_error += 1
                    with open(_file, 'a') as arq:
                        arq.write('\n[{0}] - Error -  Não foi possível inserir dados no node: {1}'.format(
                            datetime.datetime.today(), node))
            for _n in nodes_success:
                logs_saved.append({'method': 'blacklistlogs', 'action': 'insert', 'user': user, 'status': 'Success',
                                   'message': 'Dados inseridos com sucesso no node: {0}.'.format(_n)})
                with open(_file, 'a') as arq:
                    arq.write('[{0}] - Success - Dados inseridos com sucesso no node: {1}.\n'.format(
                        datetime.datetime.today(), _n))
            for _n in nodes_error:
                logs_saved.append({'method': 'blacklistlogs', 'action': 'insert', 'user': user, 'status': 'Error',
                                   'message': 'Não foi possível inserir dados no node: {0}.'.format(_n)})
                with open(_file, 'a') as arq:
                    arq.write('[{0}] - Error - Não foi possível inserir dados no node: {1}\n'.format(
                        datetime.datetime.today(), _n))

    for num in phone_numbers:
        count_success += 1
        with open(_file, 'a') as arq:
            arq.write('[{0}] - Success - Phone number {1} gravado na tabela VicidialDnc no node {2}\n'.format(
                datetime.datetime.today(), num, node))

    # Saving only unique numbers in BlackList and VicidialDnc
    if phone_numbers.__len__() > 0:
        date_init = datetime.datetime.today()
        try:
            self.serializer_class = BlackListSerializer
            _serializer = self.get_serializer(data=phone_numbers, many=True)
            _serializer.is_valid(raise_exception=True)
            self.perform_bulk_create(_serializer)
        except Exception as e:
            with open(_file, 'a') as arq:
                arq.write('\n[{0}] - {1}'.format(datetime.datetime.today(), e))
        try:
            self.serializer_class = VicidialDncSerializer
            _serializer = self.get_serializer(data=phone_numbers, many=True)
            executor.submit(_serializer.is_valid(raise_exception=True))
            executor.submit(self.perform_bulk_create(_serializer))
        except Exception as e:
            if e.args[0].__len__() > 0 and count_success > 0:
                count_repeat += e.args[0].__len__()
                count_success = count_success - count_repeat
            logs_saved.append({'action': 'insert', 'user': str(request_data[0]['user']), 'status': 'Alert',
                               'message': 'Foi identificado {0} numeros repetidos'.format(count_repeat), 'method': 'blacklistlogs'})
            with open(_file, 'a') as arq:
                arq.write('[{0}] - {1}\n'.format(
                    datetime.datetime.today(), e
                ))
        date_end = datetime.datetime.today() - date_init
        with open(_file, 'a') as arq:
            arq.write('[{0}] - Salvo o Vicidial DNC e BlackList em {1} tempo.'.format(datetime.datetime.today(),
                                                                                      date_end))

    if phones_repeat == 0 and phone_numbers.__len__() == 0 and amount_phones < phones_in_file.__len__():
        phones_repeat += amount_phones
        logs_saved.append({'action': 'insert', 'user': str(request.data['user']), 'status': 'Alert',
                           'message': 'Foi identificado {0} numeros repetidos'.format(phones_repeat),
                           'method': 'blacklistlogs'})
        with open(_file, 'a') as arq:
            arq.write('[{0}] - Foi identificado {1} numeros repetidos'.format(datetime.datetime.today(), count_repeat))

    if amount_phones > phones_in_file.__len__():
        phones_repeat += amount_phones - phones_in_file.__len__()
        logs_saved.append({'action': 'insert', 'user': str(request.data['user']), 'status': 'Alert',
                           'message': 'Foi identificado {0} numeros repetidos'.format(phones_repeat),
                           'method': 'blacklistlogs'})
        with open(_file, 'a') as arq:
            arq.write(
                '[{0}] - Foi identificado {1} numeros repetidos'.format(datetime.datetime.today(), count_repeat))

    if phone_numbers.__len__() == 0 and count_success == 0 and count_repeat == 0:
        if phones_uniques.__len__() > 0:
            date_init = datetime.datetime.today()
            try:
                self.serializer_class = BlackListSerializer
                _serializer = self.get_serializer(data=phones_unique_saved, many=True)
                _serializer.is_valid(raise_exception=True)
                self.perform_bulk_create(_serializer)
            except Exception as e:
                with open(_file, 'a') as arq:
                    arq.write('\n[{0}] - {1}'.format(datetime.datetime.today(), e))
            try:
                self.serializer_class = VicidialDncSerializer
                _serializer = self.get_serializer(data=phones_unique_saved, many=True)
                executor.submit(_serializer.is_valid(raise_exception=True))
                executor.submit(self.perform_bulk_create(_serializer))
            except Exception as e:
                if e.args[0].__len__() > 0:
                    phones_repeat += e.args[0].__len__()
                logs_saved.append({'action': 'insert', 'user': request.data['user'], 'status': 'Alert',
                                   'message': 'Foi identificado {0} numeros repetidos'.format(phones_repeat), 'method': 'blacklistlogs'})
                with open(_file, 'a') as arq:
                    arq.write('[{0}] - {1}\n'.format(
                        datetime.datetime.today(), e
                    ))
            date_end = datetime.datetime.today() - date_init
            with open(_file, 'a') as arq:
                arq.write('[{0}] - Salvo o Vicidial DNC e BlackList em {1} tempo.'.format(datetime.datetime.today(), date_end))
            print('vicidial_dnc and blacklist {0}'.format(date_end))

    if vicidial_dnc_logs.__len__() > 0:
        date_init = datetime.datetime.today()
        try:
            self.serializer_class = VicidialDncLogSerializer
            _serializer = self.get_serializer(data=vicidial_dnc_logs, many=True)
            executor.submit(_serializer.is_valid(raise_exception=True))
            executor.submit(self.perform_bulk_create(_serializer))

        except Exception as e:
            with open(_file, 'a') as arq:
                arq.write('[{0}] - Error - Não foi possivel inserir dados na vicidial dnc log - {1}\n'.format(
                    datetime.datetime.today(), e
                ))
        date_end = datetime.datetime.today() - date_init
        with open(_file, 'a') as arq:
            arq.write('[{0}] - Salvo o Vicidial DNC Logs em {1} tempo.'.format(datetime.datetime.today(),
                                                                                      date_end))
        print('vicidial_dnc_logs {0}'.format(date_end))

    if logs_saved.__len__() > 0 and isinstance(logs_saved, list):
        date_init = datetime.datetime.today()
        try:
            self.serializer_class = LogSerializer
            _serializer = self.get_serializer(data=logs_saved, many=True)
            _serializer.is_valid(raise_exception=True)
            self.perform_bulk_create(_serializer)
        except Exception as e:
            with open(_file, 'a') as arq:
                arq.write('[{0}] - Error - Não foi possivel inserir o log - {1}\n'.format(
                    datetime.datetime.today(), e
                ))
        date_end = datetime.datetime.today() - date_init
        with open(_file, 'a') as arq:
            arq.write('[{0}] - Salvo o Logs em {1} tempo.'.format(datetime.datetime.today(),
                                                                                      date_end))
        print('logs {0}'.format(date_end))
    if black_list_logs_saved.__len__() > 0:
        date_init = datetime.datetime.today()
        try:
            self.serializer_class = BlackListLogsSerializer
            _serializer = self.get_serializer(data=black_list_logs_saved, many=True)
            _serializer.is_valid(raise_exception=True)
            self.perform_bulk_create(_serializer)
        except Exception as e:
            with open(_file, 'a') as arq:
                arq.write('[{0}] - Error - Não foi possivel inserir dados na black list log - {1}\n'.format(
                    datetime.datetime.today(), e
                ))
        date_end = datetime.datetime.today() - date_init
        with open(_file, 'a') as arq:
            arq.write('[{0}] - Salvo o BlackList Logs em {1} tempo.'.format(datetime.datetime.today(),
                                                                                      date_end))
        print('black_list_logs {0}'.format(date_end))
    try:
        with open(_file, 'a') as arq:
            arq.write('\n[{0}] - Erros: {1} Alert: {2} Success: {3} Invalid: {4}'.format(
                datetime.datetime.today(), count_error, phones_repeat,
                phones_uniques.__len__(), phones_invalid.__len__()))
        return Response({'Erros': count_error,
                         'Alert': phones_repeat,
                         'Success': phones_uniques.__len__(),
                         'Invalid': phones_invalid.__len__()}, status=status.HTTP_201_CREATED)
    except Exception as ex:
        with open(_file, 'a') as arq:
            arq.write('\n[{0}] - Erros: {1} Alert: {2} Success: {3} Invalid: {4}'.format(
                datetime.datetime.today(), count_error, count_repeat,
                count_success, phone_invalid))
        return Response({'Erros': count_error,
                         'Alert': count_repeat,
                         'Success': count_success,
                         'Invalid': phone_invalid}, status=status.HTTP_201_CREATED)

utils.py

@classmethod
def verification_phone_number_exist(cls, model, serializer, phone_number):
    phones = model.objects.filter(phone_number=phone_number)

    if phones.exists():
        return True
    return False

@classmethod
def treating_numbers_in_file(cls, file_excel, _serializer):
    phones = []
    for rows in file_excel.iterrows():
        phones.append(str(rows[1].values[0]))
    phones_end = [phone for phone in phones if 9 < phone.__len__() < 12]
    _phones_end = [phone for phone in phones_end if OrderedDict([('phone_number', str(phone))]) not in _serializer.data]
    phones_invalid = [phone for phone in phones if 9 >= phone.__len__() >= 12]
    return _phones_end, phones_invalid, phones_end.__len__()

Django.ini (uwsgi.ini)

[uwsgi]

# master
master                  = true

# maximum number of processes
processes               = 24

# the socket (use the full path to be safe)
http          = 127.0.0.1:8001

# with appropriate permissions - *may* be needed
chmod-socket    = 664

# the base directory
chdir           = /var/www/html/backend

# Django's wsgi file
module          = api.wsgi:application
master=True

# the virtualenv
home            = /var/www/html/.local/share/virtualenvs/backend-pcI0BIcF

# clear environment on exit
vacuum          = true

env = DJANGO_SETTINGS_MODULE=api.settings

Answer 1

Python default is single thread, however there are ways to transform into multiprocess, analyzes more info here and here. There are several approaches and solutions, I will explain the one I used.

I recently developed a streaming solution that entails making somewhat high processing cost calculations on an M2m architecture, after a POST of a service happened the same as you describe, while processing blocked the service. My choice was to create tasks with Celery and Rabbitmq, you got a example. That is, in your create() function, you create a task and inform the user that the upload will be processed in the background and as soon as you finish you inform again that it is over, you will have to create a message service for this process. The installation will become more complex a little, but with the advantage of being able to parallelize the processing.

A part of my example:

In your create() function, you call the function, which in your case will have almost all the data upload code:

 sensor_exists_errors.delay()

And this call from the function with the . delay, will send a message to Broker, Rabbitmq, (as I suggested but you can use Redis), which is to process, the file name for example and the task id, the available Workers will process the upload as soon as possible.... this is a summary of the flow...any additional issue disposes.

@task
def sensor_exists_errors():
    cpu = psutil.cpu_percent()
    if cpu < 60:
        return 'Done ERRORS validation sensor_exists_errors()'
    return 'CPU Busy sensor_exists_errors()'

NOTE: With Celery, you can additionally create tasks with delay, as if they were Crone Jobs, you can even validate the CPU status and not process and perform the processing later that there is more CPU availability.