1
I receive daily an Oracle database with 12 tables containing on average 8 million records each.
I need to treat it to get only the updated day records compared to the previous day’s base, but I don’t have any field like "updated", which indicate which records to handle.
I just need to do it a little faster.
I’m studying some approach with Pentaho Data Integration, Elasticsearch and the next would be something with Mongodb, but I haven’t reached a concrete solution yet.
Could someone indicate me tool or method, to work this data volume?
Reading over the experience that I have would be something to generate a history can be even in a database in the nosql containing a composite key containing the date + primary key of the item in question that generates the value of the change. I believe that it can kill two rabbits with one stick and in addition at the time of reading it does not link reading to important tables for business.
– user157494
The basis of the day has the records of the previous day ? Each day dd 12 mi is saved ?
– Motta
Yes. Every day a dump of the original tables is generated and made available to me.
– Marcelo Barros da Silva