Modifying a Python file without losing the current content

Question

That doubt is a continuation of this here.

I have a very large TXT file (about 6GB), each line being 1,300 characters, and I am manipulating these lines, as an example, I will use these three lines here (the site is eating the blanks, but the lines have the same layout):

123456 BANANA 00 SP
123457 MACA   01 RJ
123458 PERA   02 MG

What I need is if the line contains the word "BANANA" at 8:14, it changes the "00" from 16:17 to "22".

For that, I’m implementing something like:

arquivo = open('testando.txt', 'r') #abre o arquivo
for linha in arquivo:
    codigo = str(linha[8:14])
    
    if codigo == 'BANANA':
       print("ACHOU, CODIGO: " + codigo)
       encontrado = True
    else:
       print("NÃO ACHOU, CÓDIGO: " + codigo)
       encontrado = False

    if encontrado:
        new_line = linha[:16] + "22" + linha[17:] #como python utiliza string imutáveis, estou apenas remontando ela
        print(new_line)
        linha = linha.replace(linha, new_line) #substituo a linha antiga pela nova, com a nova informacao
        arquivo = open('testando.txt','w')
        arquivo.write(linha) #adiciono minha nova linha no arquivo
        
    print(linha)
        
arquivo.close()

The problem I am encountering is that it is only saving the line that I changed and deleting all the rest of the file. I imagined that I was going through line by line, it was going to change only the line of that index, however it is replacing the complete file.

PS: I have tried the method write after the if and I’ve also tried writelines.

Know somehow to keep the file intact and change only the desired part?

Answer 1

First, call str in str(linha[8:14]) is redundant and unnecessary as linha[8:14] already returns a string. In addition, in the example file you reported, this excerpt corresponds to "ANANA " (but let’s assume that in the original file the indexes are the same, is a small adjustment that does not interfere with the rest).

Another point is that you are reading from the file at the same time you write to it. If there is any error in the middle of the way, the file will get corrupted, then ideally you first write everything in another temporary file, and only at the end, if all goes well, move the temporary file to the original.

And as I already said in your previous question, this variable encontrado is unnecessary. If you want to do something if the string is "BANANA", do everything within the first if:

import shutil, tempfile

# lê do arquivo e escreve em outro arquivo temporário
with open('testando.txt', 'r') as arquivo, \
     tempfile.NamedTemporaryFile('w', delete=False) as out:
    for linha in arquivo:
        codigo = linha[8:14]
        
        if codigo == 'BANANA':
           print("ACHOU, CODIGO: " + codigo)
           linha = linha[:16] + "22" + linha[17:] # remontar a linha
        else:
           print("NÃO ACHOU, CÓDIGO: " + codigo)

        out.write(linha) # escreve no arquivo temporário

# move o arquivo temporário para o original
shutil.move(out.name, 'testando.txt')

Note that the write stays out of if, since it is something that must always be done. The only thing that changes is that the linha is modified if you fall into if. If the code is not "BANANA", the line is written without modification.

I used with to open the files as this ensures that they will be closed at the end (even in case of error, which does not happen when you call close() directly - unless it is a block finally).

I also use the module tempfile to create the temporary archive and shutil.move to rename the file at the end.

See also that the replace that you were doing is not necessary, you can assign the reassembly in the variable itself linha.

Answer 2

As the most efficient answer in this case is substantially different from the accepted answer, I will write a little. It has already been commented in the comments that this would not be by far the appropriate structure for a mass of data of this size - yet another one that needs to be changed.

How the lines have size fixed this is possible. But you should open the file in special mode "Rb+" - if you try to open in "w" mode, the system erases the entire file even - and then, take some care to record each line back in the place where it is.

Accessing the records in a structured way:

First, however, let’s see how to access the data within of each row so that it is possible to maintain the system, without having to, at each modification, keep counting on the fingers in which column each field goes, and put it in an if.

Now, wanting to do this by playing index within each mbutidos line on multiple "if" commands on the path will make it harder - this is a typical case where you can use Python’s ability to customize access to attributes in a class to create something well legal: a class in which you access and modify each column by the field name, and internally it keeps the data in a single string, which can be written to the original or printed file.

The answer in How to extract information from a 'cnab' file using python? covers how to create a class of these -the "Field" and "Base" classes the way they are there create an object that is a "descriptor" (Field class): it customizes access to the data in attributes = and the "Base" class has the rest of the machinery to allow access to the fields:

class Campo:
    def __init__(self, inicio, final):
        self.inicio = inicio
        self.final = final

    def __set_name__(self, owner, nome):
        self.nome = nome

    def __get__(self, instance, owner):
        if not instance:
            return self
        return instance.dados_brutos[self.inicio: self.final]

class Base:
    def __init__(self, dados):
        self.dados_brutos = dados

    def __repr__(self):
        campos = []
        for name, obj in self.__class__.__dict__.items():
            if isinstance(obj, Campo):
                campos.append((name, getattr(self, name)))
        return "\n".join(f"{campo}:{conteudo}" for campo, conteudo in campos)

with this section of less than 25 lines, it is now possible to represent the lines in your file as a specific class:

class Frutas(Base):
    codigo = Campo(0, 6)
    nome = Campo(7, 13)
    valor = Campo(14, 16)
    uf = Campo(17, 19)
    

Check it out - I paste exactly the code above, plus the excerpt you passed as an example, in an interactive Python session and see how it works:

    ...: exemplo = """\ 
    ...: 123456 BANANA 00 SP 
    ...: 123457 MACA   01 RJ 
    ...: 123458 PERA   02 MG""" 
    ...:      
    ...: class Frutas(Base): 
    ...:     codigo = Campo(0, 6) 
    ...:     nome = Campo(7, 13) 
    ...:     valor = Campo(14, 16) 
    ...:     uf = Campo(17, 19) 
    ...:                                                                                                                                                                                                                       

In [34]: x = [Frutas(linha) for linha in exemplo.split("\n")]                                                                                                                                                                  

In [35]: x[0].nome                                                                                                                                                                                                             
Out[35]: 'BANANA'

In [36]: x[0].valor = "23"                                                                                                                                                                                                     

In [37]: x[0]                                                                                                                                                                                                                  
Out[37]: 
codigo:123456
nome:BANANA
valor:23
uf:SP

In [38]: x[0].dados_brutos                                                                                                                                                                                                     
Out[38]: '123456 BANANA 00 SP'

Manipulating file lines as records, with random access

Now, once a line is read, we have a way to work with it - the idea of having a class to represent a file like this open, and then using this class to read or write one line at a time, without the code that makes it need to worry about where or as the data are saved may be the most interesting. (Hence, if tomorrow you exchange the data storage elsewhere, be it an SQL database, Nosql, etc... the code that uses the data does not even need know thereof )

from pathlib import Path

class MapeadorTxt:
    def __init__(self, caminho_arquivo, classe_dados, comprimento_linha, codificacao="ASCII"):
        self.caminho = Path(caminho_arquivo)
        self.classe = classe_dados
        self.comp = comprimento_linha
        self.codificacao = codificacao
        
    def __enter__(self):
        self.arquivo = open(self.caminho, "rb+")
    
    def __exit__(self, exc_type, exc_value, exc_tb):
        self.arquivo.close()
        self.arquivo = None
    
    def __getitem__(self, index):
        if index >= len(self):
            raise IndexError
        self.arquivo.seek(index * self.comp)
        return self.classe(self.arquivo.read(self.comp).decode(self.codificacao))
    
    def __setitem__(self, index, valor):
       self.arquivo.seek(index * self.comp)
       v = valor.dados_brutos.rstrip("\n").ljust(self.comp - 1) + "\n"
       self.arquivo.write(v.encode(self.codificacao))
       
    def __len__(self):
        return self.caminho.stat().st_size // self.comp
        
    def __repr__(self):
        return f"Mapeador do arquivo {self.caminho} para a classe {self.classe.__name__}, com {len(self)} registros"
    

And with this class you can do what you propose in the question, using the above class in a "with" block-and we use the function enumerate in the for to have, in addition to the record, the index also - (is in the variablei) and then, if you want to access the log on the top line, just use m[i - 1], for example.

Working here to change the banana code to "42":

In [57]: m = MapeadorTxt("exemplo.txt", Frutas, 19)                                                                                                           

In [58]: with m: 
    ...:     for i, fruta in enumerate(m): 
    ...:         if fruta.nome.strip() == "BANANA": 
    ...:             fruta.valor = "42" 
    ...:             m[i] = fruta 
    ...:                                                                                                                                                      

In [59]: cat exemplo.txt                                                                                                                                      
123456 BANANA 42 SP
123457 MACA   01 RJ
123458 PERA   02 MG

This above code is not perfect - in particular, the "field" should handle text encoding and Decoding, and leave the internal values in bytes, or in a "bytearray".

And of course, that doesn’t solve the problem of access time - you still have 500 million records, which in a comparison of these have to be read one by one - play on an SQL basis, and you can search for dates, etc...