Python parallelization slower than serial

Question

Hi, I’m trying to learn a little Python parallelization to parallelize some loops for. I’m using the joblib. However, the first example I made, available on the library’s own website, does not seem to have worked very well. The parallelized code is absurdly slower than the serial. I’ve changed the number of processors used according to the available machine, but nothing! Ah, by default the library works with multiprocessing, due to python GIL issue, etc...

from math import sqrt
from joblib import Parallel, delayed
import time

#Serial 
start=time.time()
def f():
    results = []
    for i in range(100000):
        results.append(sqrt(i ** 2))
    return results

c=f()
print(time.time()-start)
0.07995438575744629

#Paralelo 
start=time.time()
def g():
    def _g(x):
        return sqrt(x)
    return Parallel(n_jobs=4)(delayed(_g)(i ** 2) for i in range(100000))
d=g()
print(time.time()-start)
3.191032648086548

Someone could give way to understand what is happening?

Answer 1

I don’t know "joblib" - in particular I don’t know if it creates threads or processes - but the only call to sqrt Anyway it is very small near the job of serializing an integer (in Python, integers are objects) deserialize, and do the reverse path. Serialization is required if joblib uses multiple processes. Python internally will call pickle.dumps in the parameters being passed to the other library - on "arrival", you have to call pickle.loads - after calculating the result, the reverse path. the call math.sqrt on the other hand is a thin layer around the code to compute square root at 64bit floating points straight on the CPU and is much quick.

If it, on the other hand, uses multi-threading, you get nothing, because in Python only one thread uses the CPU at a time, and everything runs on a single kernel (you get gains if you do I/O - write to files, or network access to threads) - but would not need the data serialisation process. For the longer delay, I’d say he’s using multiple processes.

With a more realistic Workload, you can see some gains there with joblib - including assessing if you need it or if you can use it concurrent.futures which already comes along with Python.

In this answer I have more considerations about Python parallelization: Python serial vs multiprocessing vs threading code