How to read a large file row by row with Javascript (nodejs)

Question

I have a very large file to import data to Mongodb via Javascript. The problem is that I cannot load the entire file into memory.

I would like to read this file line by line, since each line is a record to be entered in my database. I know the Fs library also works with stream, but it only searches for predefined byte sizes, without worrying whether the line is over or not. Someone’s already done it?

Answer 1

There is an interesting project on Github designed specifically to deal with this problem:

Line Reader

Asynchronous line-to-line file reader.

Example:

var lineReader = require('line-reader');

lineReader.eachLine('file.txt', function(line, last) {
  console.log(line);

  if (/* done */) {
    return false; // stop reading
  }
});

The function eachLine reads every line of the given file. On top of each new line, the return function is called with two parameters: the read line and a boolean value that specifies whether the read line was the last line of the file. If the callback returns false, the reading will stop and the file will be closed.

Answer 2

Using Nodejs native methods you have at least two options.

Using readline

var rl = readline.createInterface({
      input : fs.createReadStream('/path/file.txt'),
      output: process.stdout,
      terminal: false
})
rl.on('line',function(line){
     console.log(line) // aqui podes fazer o que precisas com cada linha
})

Using Fs.readFile()

In this case you will read the whole file first and then break that content by jigsaw puzzle.

fs.readFile('/path/file.txt', 'utf-8', function(err, data){
    var linhas = data.split(/\r?\n/);
    linhas.forEach(function(linha){
       console.log(linha); // aqui podes fazer o que precisas com cada linha
    })
})