Opening and reading giant C files

Asked

Viewed 896 times

-1

How can I open a file for reading in C language, above 100 MB?

With this code placed below, I can read a file with more than 18,000 lines, but what I really need is to read a file with approximately 5,000,000 ( 5 million lines ).

This is possible in C?

char **xxxx;
    if ( ( arq = fopen("Meuarquivo.txt", "r+b" ) ) == NULL ){
        printf("Houve um erro na abertura do arquivo");
        getche();exit(1);
    }
xxxx = ( char** ) malloc ( 19000 *sizeof ( char* ) );
    while ( feof ( arq ) == 0 ){
        nomes [ c ] = ( char* ) malloc ( 19000 *sizeof ( char ) );
        fgets ( nomes [ c ], 19000, arq );
        ++c;
    }
c filing-cabinet
  • 2

    5,000,000 lines with 19,000 characters each gives more than 88Gb of RAM. 5,000,000 lines with 80 characters each gives approximately 400Mb of RAM.

  • No friend, I may have put wrong, by lines should be around 100 characters

  • 1

    First of all, are you sure you need to use the whole file in memory at once? Normally if you are going to process the file for something else, you can read and work on the data sequentially, there is no reason to store in memory what has already been processed.

  • I can’t read the file by parts, I’ve tried limiting start and end with for loop, but it only works in a whole file read

  • Put all your code that reads. See MVCE. Say what happens when you try to read the big file. Error? Which one? Tested to see which size starts giving error?

  • Not that I’m recommending but you already tried to read the whole file at once?

  • Of course I tried, over and over again. I’ll tell you something, I created a code here, where I read a file by parts, chosen in the terminal, sine start and end, I present the total of lines read in the choice and then I do a search, and it works perfectly, but there are only 18,178 lines, What is this near and almost 5,000,000 lines? well I wish I could do the same with this bad code doesn’t work

  • 1

    You don’t pass on important information to help you, saying it doesn’t work doesn’t give subsidies to anyone to help you. Do what I told you. I’d like to answer, but I don’t have anywhere to start with so little information. I can pass a solution and you say it still doesn’t work. Even if it works for me.

  • @Samuellima what operating system and compiler do you use? How much you can allocate depends on a lot of things.

Show 4 more comments

1 answer

-1

char **xxxx;
if ((arq = fopen("Meuarquivo.txt", "r+b")) == NULL) {
    printf("Houve um erro na abertura do arquivo");
    getche();
    exit(1);
}
int c = 0;
xxxx = malloc(5000000 * sizeof *xxxx); // 5 milhoes de linhas
if (!xxxx) /* erro */;
while (1) {
    xxxx[c] = malloc(100);             // com no maximo 99 caracteres cada
    if (!xxx[c]) /* erro */;
    if (fgets(xxxx[c], 100, arq) == NULL) /* erro */;
    ++c;
    if (c = 5000000) /* erro */;
}

// usar xxxx

for (int k = 0; k < c; k++) free(xxxx[k]);
free(xxxx);

  • I’ll test it and get back to you

  • Those "magic numbers" (5000000 and 100) should be a few #defines

  • Description Resource Path Location Type invalid Conversion from void*' to char*' main.cpp /main/src line 71 C/C++ Problem

  • Oops ... you are not using a C compiler (or possibly the C language). I cannot help you with C++ because I do not know this language.

  • I’ll compile on C, hold on

  • Description Resource Path Location Type Suspicious semicolon Lunac. c /Lunac/src line 85 Code Analysis Problem Suspicious semicolon Lunac. c /Lunac/src line 88 Code Analysis Problem Suspicious semicolon Lunac. c /Lunac/src line 89 Code Analysis Problem Suspicious semicolon Lunac. c /Lunac/src line 91 Code Analysis Problem Until it compiled despite warnings, but closed the windows terminal

  • Us ifThat’s where I put it /* erro */; replaced by an appropriate error treatment.

  • Windows closes the program well before reading the file, but it was worth many thanks

Show 3 more comments

Browser other questions tagged c filing-cabinet

You are not signed in. Login or sign up in order to post.