0
While performing a function to open a very large file ,the same returned me this error
Error: Cannot create a string longer than 0x3fffffe7 characters
at stringSlice (buffer.js:602:44)
at Buffer.toString (buffer.js:668:10)
at Object.readFileSync (fs.js:385:41)
at readArq (/var/www/Split-xml-document/split.js:8:17)
at geraArquivos (/var/www/Split-xml-document/split.js:26:15)
at /var/www/Split-xml-document/split.js:63:11
at process._tickCallback (internal/process/next_tick.js:68:7)
at Function.Module.runMain (internal/modules/cjs/loader.js:744:11)
at startup (internal/bootstrap/node.js:285:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:739:3)
(node:26810) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:26810) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
The function I performed was as follows
function readArq(arq) {
let arquivo = arq.replace(".bz2", "");
//abre o arquivo wikipedia.part_20.xml
let data = fs.readFileSync(cwd + "/dataset2/" + arquivo, "utf-8");
return data;
}
The file in question is in the range of 1Gb ,so I searched ,the error is that the function can not open very large files, the question is ,there is some other function of Node js that can open large files, the file is a . xml ,or some option where I can use Fs.readFileSync stating that the limit is 1Gb or more?
What do you want to do with this file?
– Sorack
I have another function that divides him in half to facilitate reading.
– Israell Llopes
If you won’t do any processing using the file you can just use a stream to split it
– Sorack
yes ,the file need to process ,I divide it ,and as it divides exactly in half,I need to take the initial settings of the first file and copy to the second ,among other things
– Israell Llopes
You can use
streamto perform the transformation in the file you want. Readline. The ideal is that you do not read all the information at once but only check which line is and transfer to file 1 or file 2– Sorack
Are there any examples? I don’t know these functions
– Israell Llopes
It depends, we need to know exactly what you need to do with a file, an example of a small file and the outputs to know what you really need
– Sorack