Log system losing data, how to use Threads?

Question

Currently I have a medium level system where I need to record detailed log of everything that happens in the system, the Company that is divided into Departments does audit of everything that happens through the Log!

I have a Procedure very simple that makes access to the File and edits it by passing the new Log line that will be saved:

procedure frmFuncoes.GravaLogDetalhado(Texto: String);
var
  Arquivo : TextFile;
begin
  AssignFile(Arquivo, 'D:\Sistema\Log\LogDetalhado.txt');
  Append(Arquivo);
  Writeln(Arquivo, Texto);
  CloseFile(Arquivo);
end;

Well, when I need to record Log of some event I add as follows:

procedure frmDetCliente.FormCreate(Sender: TObject);
begin
  GravaLogDetalhado('Abriu Detalhes do Clientes');
  ...
  ...//Blocos e blocos de códigos
  ... 
  GravaLogDetalhado('Clicou em Editar Cliente');
  ...//Blocos e blocos de códigos
  ...
  GravaLogDetalhado('Fechou Detalhes do Clientes');
end;

The problem I face is that the Company does not want a Log for each department, but a single Log saved on the Server, it starts to complicate when the possibilities appear:

• Warehouse started launching items in the Database Dice.

• Finance began to generate payroll. ....

That one Procedure also receives an Sqlmonitor that sends the SQL command also to the Log all Insert, Update etc that can be executed in a database.

I have been reading and it seems to me that the output to avoid data loss (LOG) is to use Threads. Where to start ?

Answer 1

From what I can understand, your system must be running on more than one computer simultaneously. In such cases, trying to write file for more than one process results in data loss due to file lock. Another aggravating factor is that when a file is opened by a share it takes longer to be released for writing by another computer on the network. Even worse if it’s samba ...

So you should always use a more efficient mechanism to write log. I would use a service in style MESSAGE QUEUE to write the same log from different sources. A table in the database would solve this and would make it easier in the future to check the occurrences in the LOG.

Another way to write the same log from different sources is to use the Windows log engine that is able to write in the event log also facilitating the search.

This article is very good to see how to make a write log engine on a remote computer (server)

To write to the local system logo, you can use the code below

uses SvcMgr; // Adicione ao seu Uses 

procedure frmFuncoes.GravaLogDetalhado(Texto: String, EventType: DWord = 1);
begin
  with TEventLogger.Create('NOME_APLICACAO') do
  begin
    try
      LogMessage(Texto, EventType);
    finally
      Free;
    end;
  end;
end;

and in its code would use

procedure frmDetCliente.FormCreate(Sender: TObject);
begin
  GravaLogDetalhado('Abriu Detalhes do Clientes', EVENTLOG_INFORMATION_TYPE);
  ...
  ...//Blocos e blocos de códigos
  ... 
  GravaLogDetalhado('Clicou em Editar Cliente', EVENTLOG_INFORMATION_TYPE);
  ...//Blocos e blocos de códigos
  ...
  GravaLogDetalhado('Fechou Detalhes do Clientes', EVENTLOG_INFORMATION_TYPE);
end;

This way you pass the writing control to the System.

Answer 2

I have long needed a solution to something similar to your problem, not so big as but good looking! The solution however was to use several Thread to solve, as I am not Delphi developer I had to buy a system and pay for its maintenance!

It took me a while to post the answer since I edited this question because I was looking for a contact from an old friend who solved the mentioned problem!

Follow the implementations he made for me!

Unit unt_funcaoLog

type
  {Classe responsável pela geração de LOG!}
  TGeraLog = Class
    private
      FCritical : TCriticalSection;

      Const C_ARQUIVO = '.\arquivo_log.txt';
      {Indica o arquivo em que o LOG será gerado, no caso, no mesmo diretório do executável!}
      Class var FInstance : TGeraLog;
      {Instância única do objeto!}
      Class function GetInstancia: TGeraLog; static;
      {Método responsável por instanciar o objeto singleton!}
    public
      procedure AfterConstruction; override;
      procedure BeforeDestruction; override;
      procedure GeraLog(Const AReferencia: String; Const AData: TDateTime; Const ATextoLog: String);
      {AReferencia..: Referência à Thread chamadora.}
      {AData........: Data e Hora da geração do LOG.}
      {ATextoLog....: Texto a ser escrito no arquivo de LOG.}
      function TryGeraLog(Const AReferencia: String; Const AData: TDateTime; Const ATextoLog: String): Boolean;
      {Método que TENTA gerar uma nova linha de LOG!}
      {AReferencia..: Referência à Thread chamadora.}
      {AData........: Data e Hora da geração do LOG.}
      {ATextoLog....: Texto a ser escrito no arquivo de LOG.}
      Class property Instancia: TGeraLog read GetInstancia;
      {Referência à instância singleton!}
  end;

implementation

{ TGeraLog }

procedure TGeraLog.AfterConstruction;
begin
  inherited;
  DeleteFile(Self.C_ARQUIVO);
  Self.FCritical := TCriticalSection.Create;
end;

procedure TGeraLog.BeforeDestruction;
begin
  inherited;
  Self.FCritical.Free;
end;

procedure TGeraLog.GeraLog(const AReferencia: string; const AData: TDateTime; const ATextoLog: string);
var
  _arquivo   : TextFile;
  sNovaLinha : String;
begin
  sNovaLinha := Format('%s|%s|%s', [AReferencia, DateTimeToStr(AData), ATextoLog]);

  {Entra na seção crítica!}
  Self.FCritical.Enter;
  try
    AssignFile(_arquivo, Self.C_ARQUIVO);
    if (FileExists(Self.C_ARQUIVO)) then
    begin
      Append(_arquivo);
    end
    else
    begin
      Rewrite(_arquivo);
    end;

    Writeln(_arquivo, sNovaLinha);

    CloseFile(_arquivo);
  finally
    {Sai da seção crítica}
    Self.FCritical.Release;
  end;
end;

Class function TGeraLog.GetInstancia: TGeraLog;
begin
  if not(Assigned(FInstance)) then
  begin
    FInstance := TGeraLog.Create;
  end;
  Result := FInstance;
end;

function TGeraLog.TryGeraLog(const AReferencia: String; const AData: TDateTime;
  const ATextoLog: String): Boolean;
var
  _arquivo   : TextFile;
  sNovaLinha : String;
begin
  sNovaLinha := Format('%s|%s|%s', [AReferencia, DateTimeToStr(AData), ATextoLog]);

  {Tenta entrar na seção crítica!}
  Result := Self.FCritical.TryEnter;
  if (Result = True) then
  begin
    try
      AssignFile(_arquivo, Self.C_ARQUIVO);
      if (FileExists(Self.C_ARQUIVO)) then
      begin
        Append(_arquivo);
      end
      else
      begin
        Rewrite(_arquivo);
      end;

      Writeln(_arquivo, sNovaLinha);

      CloseFile(_arquivo);
    finally
      {Sai da seção crítica}
      Self.FCritical.Release;
    end;
  end;
end;

initialization

finalization

  TGeraLog.Instancia.Free;

end.

Unit unt_diversasThreads

type
  TDiversaThread = Class(TThread)
    private
      FReferencia    : String;
      FDescricaoErro : String;

    public
      procedure Execute; override;
      {Rotina a ser executada pelo Thread que eventualmente gerará uma linha no arquivo de LOG!}

      property Referencia: String read FReferencia write FReferencia;
      {Referência que será escrito no arquivo de LOG para sabermos de onde veio a linha!}

      property DescricaoErro: String read FDescricaoErro;
  end;

implementation

{ TDiversasThread }

procedure TDiversaThread.Execute;
var
  bGeraLog : Boolean;
begin
  inherited;
  try
    {Loop enquanto o Thread não for finalizado!}
    while not (Self.Terminated) do
    begin
      {Aqui definimos um time para diminuir o consumo de CPU}
      Sleep(10);

      {Sorteia um número e verifica se o resto da divisão por dois é zero!}
      bGeraLog := (Random(1000000) mod 2) = 0;

      if (bGeraLog = True) then
      begin
        {Chama o método de geração de LOG!}
        TGeraLog.Instancia.GeraLog(Self.FReferencia, Now, 'O rato roeu a roupa do Rei de Roma');
      end;
    end;
  except
    on E: EInOutError do
    begin
      Self.FDescricaoErro := Format('Erro de I/O #%d - %s', [E.ErrorCode, SysErrorMessage(E.ErrorCode)]);
    end;
    on E: Exception do
    begin
      Self.FDescricaoErro := Format('(%s) - %s', [E.ClassName, E.Message]);
    end;
  end;
end;

end.

I’m recording the log on this call:

TGeraLog.Instancia.GeraLog('QUEM_ENVIOU_LOG', Now, 'TEXTO_QUE_DESEJA');

According to my friend the function Trygeralog is more efficient when there is multiple access to the same file, but as it is not my case I use only Geralog.

Answer 3

The best solution in this case is to develop a separate system that listens to a port via UDP on the computer where the log file is. When receiving a log request, this application puts this request in a queue, while a Ttimer checks the list from time to time and saves the requests in the correct order.

In your main application you place a precedent that makes this call to the log application via UDP by the server IP. It is a simple alternative and it will end up with all the file lock and data loss issues.

Answer 4

I don’t believe that Threads for each event would be a solution, since several departments will be able to access the same log file at the same time. I believe there were other problems.

The best output in my opinion would be for each department to create its own log during a process and at the end of this process a thread is created to mount a general log using the data from this local log.

It would be more or less in that order:

1 - Process initiated by the user and all actions saved in a local file.

2 - At the end of the process create a thread for exclusive access to the log file "global".

An example would be

begin
  if FileExists(Filename) then
    fstm := tFileStream.Create(Filename, fmOpenReadWrite or fmShareExclusive)
  else
    fstm := tFileStream.Create(Filename, fmOpenReadWrite or fmShareExclusive or fmCreate) end;

This will ensure that only one thread will access the file.

You can control other threads to wait for file access.

Answer 5

I believe that a perhaps even simpler solution is to write this log in a table in the main database, or even in a separate database, depending on the amount of data that will be generated by these logs. From what I understand it would be multiple users running their executable on different computers, right? In this case, using threads to protect access to the file would not help much, no.

Answer 6

I do not advise the use of Threads (my opinion), after using several I come to the conclusion that it is not worth it because you can not debug is when sometimes stop working without a logical explanation, Even better is you have a Unit of functions where the processes are being done without harming the manipulation of the open screen. I’m posting an example of how you could solve your problem.

procedure TForm1.Button2Click(Sender: TObject);
begin
 GravaLogDetalhado('Almoxarifado','Abriu Detalhes do Clientes');
end;

procedure TForm1.GravaLogDetalhado(Departamento,Texto: String);
 var
  Arquivo  : String;
  BkpLog   : TextFile;
begin
  Arquivo  := 'C:\Test.txt';

 if not(FileExists(Arquivo)) then
    begin
    AssignFile(BkpLog,Arquivo );
    //Um novo arquivo em disco com o nome atribuído à variável "Arquivo" é
    //criado e preparado para processamento, um ponteiro de arquivo é colocado no início do arquivo criado.
    //Caso o arquivo a ser criado já exista, esta instrução apaga o arquivo para criá-lo novamente.
    Rewrite(BkpLog);
    CloseFile(BkpLog);
     end
  else
    begin
        AssignFile(BkpLog, Arquivo);
      //Abre um arquivo texto existente para estendê-lo (saídas adicionais). Caso o arquivo não exista,
      //esta instrução provocará um erro de Entrada/Saída (Input/Output).
        Append(BkpLog);
        Writeln(BkpLog, Departamento + ' - ' + DateTimeToStr(now) + ' : ' + Texto );
        CloseFile(BkpLog);
  end;
end;