TO BE executed asynchronously

Asked

Viewed 98 times

2

Is there any way to execute a FOR asynchronously?

What happens is this, my program recovers a large number of data from the database, processes it and returns the result to the user. I’m doing it this way:

queryset = Model.objects.raw(seleciono os dados que quero do banco)

for res in queryset:
    processamento_dos_dados

return resultado_do_processamento

The problem is that a large volume of data is often recovered, causing it to take a certain time until a return is given to the user. I have no way to modify the data processing, so the question arose whether it is possible to run this FOR asynchronously.

Researching about it, I found about the Celery and Asyncio, however, all examples are with the use of functions, and not with loopholes.

python django django-rest-framework

  • Willian, the subject is a little complex but for you to be able to solve your problem it is necessary to understand that the module asyncio does not parallelize your code directly on your Cpus, what it does is a "relay" to perform something while waiting for the response of an IO - Input/Output operation (such as writing to HD, network card response, etc...). I mention that because if processamento_dos_dados is something that will be processed in the CPU you will not take advantage of this module.

  • 1

    Celery can be a good option because you can have different processes processing your data or even different computers, but you need to assess whether an optimization in your data processing would no longer solve it. Anyway, I apologize for not formulating a real answer, but despite knowing the concept I have not implemented anything with Celery to be able to exemplify for you.

  • @fernandosavio So man, regarding processing, algotimo has already been made to be as efficient as possible. The big problem is really the amount of data it loads, sometimes it’s a few lines, sometimes it can reach thousands. Alas, as you have to process line by line, it takes a long time. I’ll take a closer look at Celery and see if I can use it to solve my problem. Thank you very much! =)

1 answer

2


Yes, there is a way to execute the for asynchronous see this article official documentation, but will not work in a queryset of Django for being synchronous, here are some suggestions for your problem

  • Try to optimize your queries
  • Refactor your algorithm using generators can greatly improve performance
  • Use Celery to orchestrate these heavier tasks

Browser other questions tagged python django django-rest-framework

You are not signed in. Login or sign up in order to post.