Pandas convert epoch str to datetime

Question

Good afternoon, everyone.

I need help converting types in pandas.

Follow the information:

I have CSV containing the Intel, a Bulletin field, ctime, mtime and the name of the files respectively, containing thousands of records as the example below:

index, iscssmodel, ctime, mtime, files
1, False, 1578221334.3724594, 1564458458.1111684, /home/brito/projetos/ccivil_03/index.htm
3, False, 1581096728.6894126, 1581096728.6894126, /home/brito/projetos/ccivil_03/pl.htm
2, False, 1578221334.3724594, 1562505992.6168315, /home/brito/projetos/ccivil_03/index_AGU.htm
7, None, None, None, /home/brito/projetos/ccivil_03/index_ÁGUA.htm

And I’ve tried the following implementations:

import time
import inspect
import pandas as pd

df = pd.read_csv(rel, na_values=['None'])

# Remove spaces on title
df.columns = [x.strip() for x in df.columns]

# Remove spaces on content
df['files'] = df['files'].str.strip()
df['mtime'] = df['mtime'].str.strip()
df['ctime'] = df['ctime'].str.strip()
df['iscssmodel'] = df['iscssmodel'].str.strip()

# sorted by filenames
df.sort_values(by=['files'], inplace=True)

# Convert epoch to datetime tz Brazil
# erro # df.ctime = pd.to_datetime(time.ctime(df.ctime))
# erro # df.ctime = pd.to_datetime(time.ctime(df.ctime.apply(int)))
# erro # df.ctime = pd.to_datetime(time.ctime(df.ctime.apply(float)))
# erro # df.ctime = pd.to_datetime(time.ctime(df.ctime.notna().apply(float)))
# Este abaixo funciona, entretanto converte boleano para segundo 1 do epoch...
df.ctime = pd.to_datetime(df.ctime.notna(), unit='s', utc=True)

# write output csv
outputfile = f"{inspect.stack()[0][3]}-{int(time.time())}.csv"
df = df[['ctime', 'mtime', 'iscssmodel', 'files']]
df.to_csv(outputfile, index=False)

How can I resolve this situation? I thank you in advance!

Answer 1

I found this solution:

...
# Tratamento de elementos Nulos
df.fillna(value=float('nan'), inplace=True)
df.replace('None', float('nan'), inplace=True)

# Convert epoch to datetime tz Brazil
df.mtime = df[df.mtime.notna()].mtime.apply(lambda x: pd.to_datetime(time.ctime(float(x))).tz_localize('America/Sao_Paulo'))
...

Where there was 'None', or '', replaced by 'Nan', then Filtrei by mtime 'notna', applying an anonymous function, which converts str epoch to datetime with Timezone -3H.