How to use joblib in Python for parallelism?

Question

I was trying to use the Thread Python to parallelize my code but I ran into a problem: when I create the Threads, the number of them exceeds 1,000 Threads easily, which, from 140, all start to give error. Researching a bit I found the joblib, but I haven’t found any example of how to use my functions... For example, I want a function, created by me, that has 3 parameters, and that function is inside a for, that will be repeated thousands of times...

repeticoes = 10000
for i in range(repeticoes):
    minha_funcao(data[i], top, param3)

I’d wear it like this?

from joblib import Parallel, delayed
Parallel(n_jobs=4, verbose=1)(delayed(minha_funcao)(data[i], top, param3) for i in range(repeticoes))

Answer 1

Threads create a "parallel processing line" within your program - but they don’t work magic: they all use resources and do exactly as you say.

Ensure that a given program does not create more than an optimal number of threads (which depending on the nature of the program may vary - a program that is intensive in computing and processing will not benefit from more than one thread per logical core of your CPU(*) - while threads that rely on response time to network requests or peripherals may benefit from a higher number).

• Pure Python also doesn’t solve much having multiple threads if the problem is processing - only one Python thread is executed at a time, no matter how many physical cores you have, due to a feature in language implementation

Anyway, a logic like: "if the total number of threads already running is greater than 'MAX_THREADS' (a number that you evaluated to be great), just write down the task, and run it in one of the threds already created when one of the running tasks is completed". This logic is not so difficult to implement in a simple way - but it becomes complicated as we want maximum efficiency, and account for all "corner cases" and always do the "right thing".

Because of this, the standard Python library (starting with version 3.2) implements objects known as "Futures" and "threadpools" - which implement exactly this logic. I suggest reading about "Concurrent." and implement your program using concurrent.futures.ThreadPoolExecutor. (And if your problem is processing-intensive, ProcessPoolExecutor - thus all processor cores can be used)