2
I was trying to use the Thread
Python to parallelize my code but I ran into a problem: when I create the Threads, the number of them exceeds 1,000 Threads easily, which, from 140, all start to give error.
Researching a bit I found the joblib
, but I haven’t found any example of how to use my functions... For example, I want a function, created by me, that has 3 parameters, and that function is inside a for, that will be repeated thousands of times...
repeticoes = 10000
for i in range(repeticoes):
minha_funcao(data[i], top, param3)
I’d wear it like this?
from joblib import Parallel, delayed
Parallel(n_jobs=4, verbose=1)(delayed(minha_funcao)(data[i], top, param3) for i in range(repeticoes))
So the idea is to make my code work on my current machine, but it will be put into production even on a server with many cores (more than 64 in only 1 node, which maybe I will use only this one). I have a lot of file reading and writing in the bank, so I thought about the competition. I think it’s best to keep this library that you suggest?
– Wandré Veloso