IMPORT SPECIFIC EVENT - PANDAS

Question

I have a script where I loop for and run about 330 files, in order to consolidate and apply a filter at the end of the process. As a beginner, I made a few attempts but without success.

I want to understand if there is any way to filter a specific event from a file column already in import. In practice, loop and consolidate only the event to be filtered.

I can’t do it in the end as it gives a memory error as the files are heavy.

for x in tqdm(all_files):
    df = pd.read_csv(x, low_memory=False, encoding='latin1', sep=';')
    lt.append(df)
frame = pd.concat(lt, ignore_index=True, axis=0)

Answer 1

You can filter using the pandas Loc or iloc

df = df.loc[['meu_campo'] == 'texto']

this will make that dataframe only have the data referring to the filter that I added

For more information on the method use the pandas doc https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.loc.html