PHP cannot process heavy files

Question

PHP cannot process heavy files

Asked 5 years, 11 months ago

Viewed 238 times

0

I have a PHP system that reads a TXT file, and performs an REPLACE in the database for each line, it does this well with small files (About 100 lines), but when I do with giant files (Over 1 Million lines) Chrome gets a white screen and does not work. How do I run something big like this in the browser?

This is an example of a line:

2018-09-03;AALR3                                             ;0000000090; 000000000012.690000;000000000000000400;10:01:00.021;1;2018-09-03;000086402559117;000000000200197;2;2018-09-03;000086402555935;000000000200198;2;0;00000003;00000003

I use only the first 7 information, so I use this code below:

<?php

$conexao = mysqli_connect("generic_ip", "genericlogin", "generickey", "genericdatabase");

if ($_POST) {
    $count=0;
    $tmp_name = $_FILES["arquivo"]["tmp_name"];
    $ler = file($tmp_name);
    $total = count($ler);
    $controler = 0;

    foreach ($ler as $linhas) {
if($controler > 0){
        $base = $linhas;
        $corrigibase = (explode(";", $base));
        $dado0 = trim($corrigibase[0]);
        $dado1 = trim($corrigibase[1]);


        $dado2 = trim($corrigibase[2]);
        $dado2 = floatval($dado2);
        $dado3 = trim($corrigibase[3]);
        $dado3 = floatval($dado3);
        $dado4 = trim($corrigibase[4]);
        $dado4 = floatval($dado4);


        $dado5 = trim($corrigibase[5]);
        $dado6 = trim($corrigibase[6]);

        $QUERY_MORE = "REPLACE INTO genericdatabase.generictable (Data, Papel, N_neg, Preco, Quant_Neg, Hora, Ind_anul) VALUES ('$dado0', '$dado1', '$dado2', '$dado3', '$dado4', '$dado5', '$dado6')";

        $INSERT_BASE = mysqli_query($conexao, $QUERY_MORE);

$count++;
        $dado0 = NULL;
        $dado1 = NULL;
        $dado2 = NULL;
        $dado3 = NULL;
        $dado4 = NULL;
        $dado5 = NULL;
        $dado6 = NULL;
}
$controler++;

    }

    echo $count."registros contados";

}

?>

The screen gets bankroll due to delay so the server can run all lines... The time depends on the machine’s processing capacity. But some error happens?

– edson alves

2018/11/28 at 12:51
The function file will read the whole file and put to mémoria at once, so one thing that may be happening is you don’t have enough memory to read this, by default php comes with limit of 250mb. If it’s really that you can use fopen, here has an example of how to use the function to process line by line.

– fajuchem

2018/11/28 at 12:52
@edsonalves, no other mistakes happen, only this

– Geraldão de Rívia

2018/11/28 at 12:54
@fajuchem, the file size is about 270 Mb, so it makes sense what you say, I didn’t put the question, but this file is imported by the user, as would use fopen in this case?

– Geraldão de Rívia

2018/11/28 at 12:56
@Geraldãoderívia Someone already gave an example of how to use fopen in the answers, besides I put a link of the comment above. It is also worth remembering that it is possible to increase the memory limit of php to make sure that this is really the problem. If you are using xampp here shows how to increase memory.

– fajuchem

2018/11/28 at 13:05
@fajuchem, tried with way q was recommended above below and msm so it didn’t work, it worked with smaller files, but with big file gives the same problem

– Geraldão de Rívia

2018/11/28 at 13:28

Show 1 more comment

1 answer

Browser other questions tagged php mysql google-chrome txt

You are not signed in. Login or sign up in order to post.

by jbrunoxd • **775** points · Answer 1 · 2018-11-28T12:55:37+00:00

Yes, with 1 million lines it is already necessary to do the operations per stream line by line, the main modification would be the following lines:

//abrir com fopen ou invés do file()
$ler = fopen($tmp_name, 'r');

//usar o while com fgets ao invés do foreach do seu código pra percorrer
//cada linha separadamente, não carregando todo o arquivo em memória
while ($linha = fgets($ler)) { ... }

//fecha o arquivo depois do while
fclose($ler);