Alternatives for storing and analyzing data

Question

A part of a project that I’m developing is to acquire voltage and current data from a solar plate. Per day I have around 40 measures like these that are here.

14/dez  16:54   18.21   0.12    18.35   0.0641
14/dez  16:54   17.82   0.27    17.98   0.2041
14/dez  16:54   17.42   0.44    17.61   0.3041
14/dez  16:54   14.23   0.39    15.30   0.3841
14/dez  16:54   5.92    0.63    7.45    0.6741
14/dez  16:54   4.97    0.54    6.50    0.6441
14/dez  16:54   4.56    0.48    5.91    0.5641
14/dez  16:54   3.51    0.68    4.59    0.7541
14/dez  16:55   2.32    0.60    2.97    0.6841
14/dez  16:55   2.19    0.51    2.76    0.6041
14/dez  16:55   2.13    0.44    2.67    0.5041
14/dez  16:55   0.71    0.67    0.90    0.7641
14/dez  16:55   0.61    0.64    0.77    0.7441
14/dez  16:55   0.00    0.69    0.00    0.821

15/dez  06:04   17.67   0.13    17.61   0.051
15/dez  06:04   15.82   0.25    15.82   0.181
15/dez  06:04   12.41   0.30    13.00   0.231
15/dez  06:04   8.09    0.22    8.72    0.161
15/dez  06:04   2.84    0.31    3.11    0.281
15/dez  06:04   2.38    0.25    2.60    0.191
15/dez  06:04   2.19    0.17    2.40    0.111
15/dez  06:04   1.69    0.36    1.79    0.301
15/dez  06:04   1.07    0.27    1.16    0.241
15/dez  06:04   1.02    0.21    1.10    0.171
15/dez  06:05   1.01    0.10    1.09    0.061
15/dez  06:05   0.33    0.32    0.35    0.311
15/dez  06:05   0.28    0.29    0.30    0.251
15/dez  06:05   0.00    0.33    0.00    0.312

I have to store this data so that in the future it is possible to chart by choosing the day and time for example.

What I want is to store and then retrieve the data more or less like this:

Ex grosseiro para recuperar: plot(data['14/12'].hora['07:00'].medidas)

Does anyone know if you can do this using a database? Or suggests another solution to store?

If it’s in Python it would be even better.

Thank you

Answer 1

For this example I put (stored) exactly the data you put in a file and went through it and removed the data, this file being (tests.txt) is in the same folder as the file with the code, you can also put (store) in a ficheiro.csv:

dados = {}
with open('tests.txt', 'r') as f:
    for line in f:
        params = [i.strip() for i in line.split('  ') if i.strip() != '']
        if not params:
            continue
        data, hora, v1, l1, v2, l2 = params
        dados[data] = dados.get(data, {})
        dados[data][hora] = dados[data].get(hora, [])
        dados[data][hora].append((v1, l1, v2, l2))

In this line [i.strip() for i in line.split(' ') if i.strip() != '']. Note that if you could it would be much simpler to separate the data per line for just ";" that would be: [i.strip() for i in line.split(';')] instead of inconstant spaces such as pus, and no empty lines could also be removed, if not params: continue.

However this code is fully functional for the data/format you placed

You can access every day and/or hour like this:

Day:

print(dados['15/dez'])

Output:

{'06:05': [('1.01', '0.33', '0.28', '0.00'), ('0.10', '0.32', '0.29', '0.33'), ('1.09', '0.35', '0.30', '0.00'), ('0.061', '0.311', '0.251', '0.312')], '06:04': [('17.67', '0.13', '17.61', '0.051'), ('15.82', '0.25', '15.82', '0.181'), ('12.41', '0.30', '13.00', '0.231'), ('8.09', '0.22', '8.72', '0.161'), ('2.84', '0.31', '3.11', '0.281'), ('2.38', '0.25', '2.60', '0.191'), ('2.19', '0.17', '2.17', '2.40'0.111'), ('1.69', '0.36', '1.79', '0.301') ('1.07', '0.27', '1.16', '0.241'), ('1.02', '0.21', '1.10', '0.171')]}

Time (here you can access the various measures of each hour):

print(dados['15/dez']['06:05'])

Output:

[('1.01', '0.33', '0.28', '0.00'), ('0.10', '0.32', '0.29', '0.33'), ('1.09', '0.35', '0.30', '0.00'), ('0.061', '0.311', '0.251', '0.312'')]

Here I will leave an alternative using defaultdict:

from collections import [defaultdict][1]

dados = defaultdict(dict)
with open('tests.txt', 'r') as f:
    for line in f:
        params = [i.strip() for i in line.split('  ') if i.strip() != '']
        if not params:
            continue
        data, hora, v1, l1, v2, l2 = params
        if hora not in dados[data]:
            dados[data][hora] = []
        dados[data][hora].append((v1, l1, v2, l2))
print(dados)

Answer 2

Yes, but I think it’s unnecessary to use a database, since you don’t seem to have much experience with it.

Search over CSV, you can store your data in a saved . txt within the same project, when you need it again just reopen csv. There are several python libraries to help you manage CSV files, it’s the easiest way to store small data.

If you want a database you can use Firebase, it works by cloud and is very simple to use. For local banks recommend Sqlite.