How to optimize memory consumption using Doctrine

Question

I need to read and save in the database two files the first (smallest) has about 49,000 lines (each line will be become a record). I parse this file by creating an instance for the object and then save that object to the database using Doctrine. I can save about 13,000 records, after which the script dies due to memory consumption.

Although the file is large having more than 100MB I don’t carry the whole file in memory, I do read it one line at a time. The complete process of reading the file, creating instance, parse and uploading all the other data for the file to work (with the exception of Doctrine) at no time consumes more than 30 MB (consumes less than 2.5% ram). Just when Doctrine runs that memory consumption increases absurdly.

I know it is possible to increase the amount of memory to be used by changing php.ini directives or using ini_set, but I would not like this solution. The object only needs to be written to the database and not to remain in memory.

I tried to recover the elements and remove from memory using detach:

private function clearBatch() {
        $id = $this->last->getId() - 1000;

        $query = $this->getDoctrine()->getEntityManager()->createQuery(sprintf('SELECT c FROM {MyObjet} c WHERE c.id >= \'%d\'', $id > 0 ? $id : 0));
        $cache = $query->getResult();
        foreach ($cache as $c) {
            $this->getDoctrine()->getManager()->detach($c);
        }
        $query->free();
        $this->getDoctrine()->getManager()->clear();
        $this->getDoctrine()->resetManager();
        gc_collect_cycles();
    }

but the script always dies, I believe that Doctrine is every Insert storing it in memory, only that I just need to feed the database with that information. There’s no need to remember.

Every 1000 records I enter (persist) I run clearBatch

public function onProcess($c){
      $this->getDoctrine()->getManager()->persist($c);
      $this->count++;
      $this->last = $c;
      if ($this->count % 1000 == 0) {
           $this->getDoctrine()->getManager()->flush();
           $this->clearBatch();
      }
}

I even tried to appeal by calling the Garbage Collector.

If I don’t store in persistence I can read all the records. And my memory consumption is constant (reaches that peak, right at the start of the script and remains until the end) 2.4% (is a Vmware only for testing, has 1GB of ram), the script dies when after reaching ~14%.

The function onProcess is called in the file read loop.

Answer 1

The problem is precisely in the file size, only counting by the number of lines the file should be more than 50M. Then there’s not much to do your memory consumption will be high even.

I suggest two things or you make a script just to do this processing in pure PHP, consumption will still be great but then you can optimize better.

If you want to continue using Doctrine you first try to leave the entityManager in a global access property/variable ($this->em), start in the constructor for example, instead of always giving a $this->getDoctrine->getManager()

Something else tries to use only the clear in the onProcess function

public __contruct() {
    //enable gc
    gc_enable();

    //get entityManager
    $this->em = $this->getDoctrine()->getManager();

    //disable log (salva um pouco de memoria ao evitar que tudo vá para o log)
    $this->em->getConnection()->getConfiguration()->setSQLLogger(null);
}

public function onProcess($c) {

    $this->em->persist($c);
    $this->count++;

    unset($c);

    //diminui um pouco o batch para evitar varios statements
    if ($this->count % 100 == 0) {
         $this->em->flush();
         $this->em->clear(); //detach all objetcs

         gc_collect_cycles();
    }
}

Your clearBatch method will only slow down the process.