How to use joblib in Python for parallelism?

Asked

Viewed 504 times

2

I was trying to use the Thread Python to parallelize my code but I ran into a problem: when I create the Threads, the number of them exceeds 1,000 Threads easily, which, from 140, all start to give error. Researching a bit I found the joblib, but I haven’t found any example of how to use my functions... For example, I want a function, created by me, that has 3 parameters, and that function is inside a for, that will be repeated thousands of times...

repeticoes = 10000
for i in range(repeticoes):
    minha_funcao(data[i], top, param3)

I’d wear it like this?

from joblib import Parallel, delayed
Parallel(n_jobs=4, verbose=1)(delayed(minha_funcao)(data[i], top, param3) for i in range(repeticoes))

1 answer

1

Threads create a "parallel processing line" within your program - but they don’t work magic: they all use resources and do exactly as you say.

Ensure that a given program does not create more than an optimal number of threads (which depending on the nature of the program may vary - a program that is intensive in computing and processing will not benefit from more than one thread per logical core of your CPU(*) - while threads that rely on response time to network requests or peripherals may benefit from a higher number).

  • Pure Python also doesn’t solve much having multiple threads if the problem is processing - only one Python thread is executed at a time, no matter how many physical cores you have, due to a feature in language implementation

Anyway, a logic like: "if the total number of threads already running is greater than 'MAX_THREADS' (a number that you evaluated to be great), just write down the task, and run it in one of the threds already created when one of the running tasks is completed". This logic is not so difficult to implement in a simple way - but it becomes complicated as we want maximum efficiency, and account for all "corner cases" and always do the "right thing".

Because of this, the standard Python library (starting with version 3.2) implements objects known as "Futures" and "threadpools" - which implement exactly this logic. I suggest reading about "Concurrent." and implement your program using concurrent.futures.ThreadPoolExecutor. (And if your problem is processing-intensive, ProcessPoolExecutor - thus all processor cores can be used)

  • So the idea is to make my code work on my current machine, but it will be put into production even on a server with many cores (more than 64 in only 1 node, which maybe I will use only this one). I have a lot of file reading and writing in the bank, so I thought about the competition. I think it’s best to keep this library that you suggest?

Browser other questions tagged

You are not signed in. Login or sign up in order to post.