'system.outofmemoryexception' while opening . dbf 300 mega file

Question

I try to open a file . dbf from 300mg in my code, but this error appears '$Exception. Message' threw an Exception of type 'System.Outofmemoryexception', a smaller file of 300kb is open normally, I know the file is very large, but also know that it is possible to run files of this size (300mg), follows the code:

I’m using Windows Form...

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Data.OleDb;
using System.Collections;
using System.Threading;

namespace LerDBF
{

public partial class Form1 : Form
{
    public Form1()
    {
        InitializeComponent();
    }

    private void btnConsultaPrimaria_Click(object sender, EventArgs e)
    {
        try
        {
            OleDbConnection oConn = new OleDbConnection();
            oConn.ConnectionString = "Provider = Microsoft.Jet.OLEDB.4.0; Data Source= c:\\; Extended Properties = dBASE IV; User ID=; Password=";
            OpenFileDialog openFileDialog1 = new OpenFileDialog();
            openFileDialog1.Filter = "Arquivos DBF|*.dbf";
            openFileDialog1.Title = "Selecione o arquivo DBF";

            if (openFileDialog1.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                oConn.Open();
                OleDbCommand oCmd = oConn.CreateCommand();

                oCmd.CommandText = @"SELECT * FROM " + openFileDialog1.FileName;
                DataTable dt = new DataTable();
                dt.Load(oCmd.ExecuteReader());
                ArrayList dadosComboBox = new ArrayList();
                foreach (DataColumn c in dt.Columns)
                {
                    dadosComboBox.Add(c.ColumnName);
                }
                comboBox1.DataSource = dadosComboBox;
                oConn.Close();
                dataGridView1.DataSource = dt;
            }
        }
        catch (Exception exc)
        {
            MessageBox.Show("Erro: " + exc.Message);
        }
    }

    private void btnConsultaSecundaria_Click(object sender, EventArgs e)
    {
        try
        {
            OleDbConnection oConn = new OleDbConnection();
            oConn.ConnectionString = "Provider = Microsoft.Jet.OLEDB.4.0; Data Source= c:\\; Extended Properties = dBASE IV; User ID=; Password=";
            OpenFileDialog openFileDialog1 = new OpenFileDialog();
            openFileDialog1.Filter = "Arquivos DBF|*.dbf";
            openFileDialog1.Title = "Selecione o arquivo DBF";

            if (openFileDialog1.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                oConn.Open();
                OleDbCommand oCmd = oConn.CreateCommand();

                oCmd.CommandText = @"SELECT * FROM " + openFileDialog1.FileName;
                DataTable dt = new DataTable();
                dt.Load(oCmd.ExecuteReader());
                ArrayList dadosComboBox = new ArrayList();
                foreach (DataColumn c in dt.Columns)
                {
                    dadosComboBox.Add(c.ColumnName);
                }
                comboBox2.DataSource = dadosComboBox;
                oConn.Close();
                dataGridView2.DataSource = dt;
            }
        }
        catch (Exception exc)
        {
            MessageBox.Show("Erro: " + exc.Message);
        }
    }
}


}

I’d have to use some thread to run it in the background?

Answer 1

The solution is very simple. Do not load the whole file into memory. When giving the SELECT place a clause WHERE to take only part of the .dbf and process in parts. You will probably have to create a loop to repeat the operation until you do what you want with all the data, but trying to put them all in memory will not work. Depending on the goal, you will have to develop a complex logic to treat this.

You can also use a DataView to virtualize the data. I don’t know if it works perfectly with .dbf but I believe so.

In some cases a DataReader could be a solution.

Answer 2

Add the following code to your App.config:

 <configuration>
   <runtime>
     <gcAllowVeryLargeObjects enabled="true" />
   </runtime>
 </configuration>