How to implement journaling in Python?

Question

I need to do a series of operations involving binary files (you can’t use BD), and I need to ensure that they end successfully even in the event of failure in the middle of the operation. For this, I see no way out except to implement a system of journaling manually. I started writing a code, but I’m not sure if it’s correct (i.e. if there are no cases where a failure in the middle of the task can put it in an inconsistent state - including failure in the actual writing of the Journal).

Is there anything ready in that direction? And if there is, there is some problem in my attempt to remedy below?

def nova_tarefa(args):
    refazer_journal() # Refaz o que ficou inacabado da última vez (se houver)
    with open('args.json', 'w') as f:
        json.dump(args, f) # Prepara as novas tarefas
    with open('journal.txt', 'w') as f:
        f.write('comecar\n') # Coloca as novas tarefas no journal
    refazer_journal() # Faz as novas tarefas

def refazer_journal():
    try:
        with open('journal.txt', 'r') as f:
            passos = [x.strip() for x in f.readlines() if x.strip()]
    except:
        if not os.path.exists('journal.txt'):
            passos = []
        else:
            raise
    if not passos: # Se não há nada inacabado, termina
        return
    with open('args.json', 'r') as f:
        args = json.load(f)

    # Realiza as tarefas que ainda não foram marcadas como concluídas
    for indice,tarefa in enumerate(args['tarefas']):
        if len(passos) <= indice+1:
            realizar_tarefa_indepotente(tarefa)
            with open('journal.txt', 'a') as f:
                f.write('\ntarefa concluida\n')

    with open('journal.txt', 'w') as f:
        pass # Tudo foi feito com sucesso: esvazia o journal

(Note: what is written on Journal - comecar, tarefa concluida - it is not important, if only one letter is written on the line it is already considered that the step has been successful)

Code for testing (always worked with me - including editing the journal.txt to enter a series of errors):

def realizar_tarefa_indepotente(tarefa):
    if 'falhar' in tarefa:
        raise Exception('Tarefa falhou!')
    print 'Realizando tarefa: ' + tarefa['nome']

tarefa_ok = {'tarefas':[{'nome':'foo'}, {'nome':'bar'}, {'nome':'baz'}]}
falha_3 = {'tarefas':[{'nome':'foo'}, {'nome':'bar'}, {'nome':'baz','falhar':True}]}
falha_2 = {'tarefas':[{'nome':'foo'}, {'nome':'bar','falhar':True}, {'nome':'baz'}]}
falha_1 = {'tarefas':[{'nome':'foo','falhar':True}, {'nome':'bar'}, {'nome':'baz'}]}
falha_1_3 = {'tarefas':[{'nome':'foo','falhar':True}, {'nome':'bar'}, {'nome':'baz','falhar':True}]}

Answer 1

I wrote an article that describes how to do this: http://epx.com.br/artigos/arqtrans.php and even a citizen made a Python implementation based on the article: https://pypi.python.org/pypi/acidfile/1.2.0

The basic technique is to write at least two files. First one, then the other in synchronous mode - as described in other responses.

The "spice" of the scheme is to add timestamp and a sum (hash) in the two files. This way we can know which is the most recent file - by timestamp - can check if the file is intact (by hashing).

The file system must implement journaling at least of the meta-data (usually the case, journaling of the data costs performance). This ensures that at least the folder and the file are readable. If the data is corrupted you detect by the hash.

Answer 2

• An easily visible problem in your implementation is that the way you check to see if the file journal.txt exists after reading is subject to race conditions, see How to check if a file exists using Python.

• In addition, the ideal is not to capture "Naked" exceptions, but to define the specific type of exception you expect, p.ex. IOError, OSError, etc. Likewise, when raise an exception, it is good to use one with a specific name related to the error type.

• To iterate over the dictionary, as an alternative, rather than iterate over the result of the enumerate(), you can simply iterate over the list:

  passos = [{'nome':'foo','falhar':True}, {'nome':'bar'}, {'nome':'baz'}]
  for tarefa in passos:
      realizar_tarefa_indepotente(tarefa) 
  

• The way you check if the 'fail' value works, but it is not very explicit. I would recommend switching to if 'falhar' in tarefa.items(), if you are simply checking whether it exists, or if tarefa.get('falhar') if you want to make sure that the value of tarefa['falhar'] is True.

Answer 3

Guaranteed recording of files

My first thought when it comes to ensuring file processing was the buffer data.

See what the documentation says file.flush():

Note: flush() does not necessarily write the file data to disk. Use flush() followed by os.fsync() to ensure behavior. (free translation)

I suppose tasks can have a much larger volume of data than the journal.txt. If a power drop occurs at the end of the method refazer_journal(), the Journal may have been written to disk while the task data is still in buffer.

I understand the data from Journal are sent afterward, but I do not believe there is any guarantee that the buffer will be lined up sequentially by the operating system and hardware.

Answer 4

Create a journaling based on filesystem files. I suggest using an epoch-based nomenclature... Writing and deleting items at once is always safer than manipulating them thousands of times. Journaling itself can be corrupted, since python atomicity is difficult to guarantee for long processes.