Loop to perform quantity processes in quantity

Question

I have a size 30 list in python, I need to make a loop to catch 6 item from that run list, after we finish the 6 run over 6 and so on until finalizing the list.

but I’m not finding a way to do that.

   import os
    scripts[
    'exp1.py'
    'exp2.py'
    'exp3.py'
    'exp4.py'
    [...]
    'exp30.py'
    ]
def roda_processo(processo):
    os.system('python {}'.format(processo))

execute:

from multiprocessing import Pool
from scripts import *

if __name__ == '__main__':
    num_processors = #[preciso que a execução ocorra aqui 6 em 6]

    pool = Pool(processes=num_processors)
    output = pool.map(roda_processo, processos)
    print(output)

Because I need you to only run 6 processes at a time and when you finish one run the other.

Answer 1

No need to use multiprocessing to start a process that will use os.system to run a shell (another process) which in turn will run another python process to start a script!!

This way you are running 3 processes for each job you want to perform! Very inefficient and confusing.

Instead use the module asyncio. The advantage is to be able to make use of asyncio.wait() to wait for one of the processes to end; thus it is possible to start a new process as soon as any of them ends, in any order, even if one of the processes is faster than the other.

import sys
import asyncio

scripts = [
    'exp1.py',
    'exp2.py',
    'exp3.py',
    'exp4.py',
    ...
    'exp30.py',
]

async def main():
    tarefas_rodando = set() 
    while scripts:
        # inicia até 6 scripts
        while len(tarefas_rodando) < 6 and scripts:
            script = scripts.pop()
            p = await asyncio.create_subprocess_exec(sys.executable, script)
            tarefa = asyncio.create_task(p.wait())
            tarefas_rodando.add(tarefa)

        # espera um dos scripts acabar
        finalizadas, tarefas_rodando = await asyncio.wait(tarefas_rodando, 
            return_when=asyncio.FIRST_COMPLETED)

    # finalizou, espera o restante terminar
    await asyncio.wait(tarefas_rodando, return_when=asyncio.ALL_COMPLETED)

if sys.platform == "win32":
    asyncio.set_event_loop_policy(
        asyncio.WindowsProactorEventLoopPolicy())

asyncio.run(main())