Author: Pradeep K M
Subject: Storing huge volume of data, compare and process
Posted: Mon Jan 16, 2017 5:08 pm (GMT 5.5)
Hi,
There are total 100 flat files (TAPE) with approx 1 million records in each file, created since 2009 every month. The record length is 200 bytes - let's call it as set1. Monthly, I'll be getting another flat file with same layout comprising of approx 20 thousand records - set2. I need to compare set2 with set1 based on 18 bytes key and then write the matched records into an output file.
Notes:
* It will be a monthly process. Set1 data changes every month in such a way that, the oldest file among the 100 will be out of scope and a new file will be added every month.
* Set 2 data is not a static data - keeps changing every month.
* There is no general criteria using which I could reduce/eliminate the volume of data from 100 flat files.
* DB2 is out of scope as this needs to be finished quickly. Working with DBAs and taking approvals, access etc takes quite a long time in our company.
* Will be used only in batch job.
The queries that I have are,
* How should I handle such a huge data in an efficient way in terms of storage, performance CPU Time etc.
* Do I create a single VSAM KSDS one time to store data from 100 flat files (total will be approx 100M after removing the duplicates) and then do the compare. After comparison write the output to a new file, remove the oldest data and update the new file to the VSAM. Also, I will get some scenarios where I need to update the existing records (in Case of VSAM).
* Or Is it better to use the combined TAPE file or concatenated tape files (100) instead of going for VSAM where we need storage in disk.
* If I use tape files, I feel the efficiency will be low compared to VSAM.
* Is there any method where I could split the data and work on it or is there any other better idea?
Subject: Storing huge volume of data, compare and process
Posted: Mon Jan 16, 2017 5:08 pm (GMT 5.5)
Hi,
There are total 100 flat files (TAPE) with approx 1 million records in each file, created since 2009 every month. The record length is 200 bytes - let's call it as set1. Monthly, I'll be getting another flat file with same layout comprising of approx 20 thousand records - set2. I need to compare set2 with set1 based on 18 bytes key and then write the matched records into an output file.
Notes:
* It will be a monthly process. Set1 data changes every month in such a way that, the oldest file among the 100 will be out of scope and a new file will be added every month.
* Set 2 data is not a static data - keeps changing every month.
* There is no general criteria using which I could reduce/eliminate the volume of data from 100 flat files.
* DB2 is out of scope as this needs to be finished quickly. Working with DBAs and taking approvals, access etc takes quite a long time in our company.
* Will be used only in batch job.
The queries that I have are,
* How should I handle such a huge data in an efficient way in terms of storage, performance CPU Time etc.
* Do I create a single VSAM KSDS one time to store data from 100 flat files (total will be approx 100M after removing the duplicates) and then do the compare. After comparison write the output to a new file, remove the oldest data and update the new file to the VSAM. Also, I will get some scenarios where I need to update the existing records (in Case of VSAM).
* Or Is it better to use the combined TAPE file or concatenated tape files (100) instead of going for VSAM where we need storage in disk.
* If I use tape files, I feel the efficiency will be low compared to VSAM.
* Is there any method where I could split the data and work on it or is there any other better idea?