Parallel scripts using Threading - Python

Question

I’m trying to run parallel scripts in Python, using the threading library, but I’m not being able to apply it to my case. I believe the problem is to receive my function, but even creating a list, it returns the error. I’m having trouble executing the function.

import threading
import os

def inicia_programa(args): 
  for nome_arquivo in args: 
     os.system('py -3.8 {}'.format(nome_arquivo))

if __name__ == "__main__":

    arquivos = ['Script1','Script2','Script3']

    processos = []
    for arquivo in arquivos:
        processos.append(threading.Thread(target=inicia_programa, args=('Script1','Script2','Script3)))

    for processo in processos:
        processo.start()

the error shown indicates this:

Answer 1

You don’t need threads for this. Use the module subprocess and you can start multiple processes without blocking the main thread.

In addition, the module subprocess allows you to run python only. If you monitor your list of processes you will see that the os.system runs two processes, it starts a shell to run its process, thus wasting time and resources.

Another hint is the use of the variable sys.executable - it has the path to the same python that was used to run the script - so you don’t need to change your script when using python 3.9 or any other.

import sys
import subprocess

arquivos = ['Script1.py','Script2.py','Script3.py']
processos = []

for arquivo in arquivos:
    processo = subprocess.Popen([sys.executable, arquivo])
    processos.append(processo)

# neste ponto todos os scripts estão rodando em background ao mesmo tempo. 
# Vamos esperar todos eles terminarem:

for processo in processos:
    processo.wait()