How to kill the process automatically with Popen?

Question

I am running a parallel process in python:

process = subprocess.Popen(['python',path, title, uid])

The program takes +- 1 minute to finish and runs normally. The process generates a PID that I can capture: process.pid. In one example he generated the PID 29058. I have another program that will handle these PID and verify which ended through the function:

 def check_pid(pid):                                                             

  try:                                                                        
      os.kill(pid, 0)                                                         
  except OSError:                                                             
      return False                                                            
  else:                                                                       
      return True

Which also works normally.

But even though my program I ran on Popen finish, and I’m sure it happens. The process on Ubuntu still continues running the process, without consuming memory and nothing:

My main program which I call the first command above is a Bottle server which will not finish unless I want it to. When I kill the Bottle server process this process PID 29058 also dies. What I want to know is if there are any parameters that I pass on Popen that make the process die automatically when finished and not stand still like this?

Answer 1

If you have the object created by the call to Popen (in the variable process, in the example you gave), and detect that the process task has been completed, just call the method wait in that object.

In other words: call process.wait() after the task is finished.

Since you are calling an external process from a web application, however, there are other tips on paths to follow:

You could use ProcessPoolExecutor in the module concurrent.futures - this would allow you to have a fixed external processes pool - and the Current.Utures module takes care of reusing the processes that have already performed their task and get them ready for the next.

The other way, it requires rethinking architecture more, but that’s how this kind of task is usually done in "production" in large systems: it involves using the Celery. Celery is a framework that connects different running processes through queues, but this is done in an almost transparent way: you do what seems to be a simple function call in Python, Celery puts the parameters in a queue outside the process, It is a process coordinated by the Church, external, but which can include the same file ". py" of the program calling the function is that executes the call. This external process is a "worker" - and can be either on the same server as your web app, or on any other machine on the network (thus allowing workload distribution).