What is the best way to do a very large and repetitive task?

Asked

Viewed 83 times

4

I need to take the duration of 18000+ audios, using the library audioread for each audio it takes right 300ms, ie at least 25~30 minutes of processing.

Using a system of Queue and Process, using all the available cores of my processor, I can decrease the processing average of each audio to 70ms, but this will still take 21 minutes, how can I improve it ? I would like to be able to read all audios in at least 5 minutes, remembering that I have no competition in the machine, it will run only my software, so I can consume all resources.

Code to read the duration:

def get_duration(q, au):
    while not q.empty():
        index = q.get()
        with audioread.audio_open(au[index]['absolute_path']) as f:
            au[index]['duration'] = f.duration * 1000

Code that creates the Processes:

for i in range(os.cpu_count()):
    pr = Process(target=get_duration, args=(queue, audios, ))
    pr.daemon = True
    pr.start()

In my code there’s only one Queue with some Process, and use the Manager to edit the objects.

  • Arranges the indentation of the second loop, pfv

  • @Lucas Corrigido.

  • Have a look at the Threading https://docs.python.org/3/library/threading.htmllibrary

No answers

Browser other questions tagged

You are not signed in. Login or sign up in order to post.