Just now I had an occasion to work with an integration solution intended to process lots of records. By lots I mean over 1 million smallish records. My customary platform to experiment on is Windows XP. Lots of reasons for that, most of them historical – I have tools I know and like and so on. While trying to work with such a volume of data I noticed a number of “interesting” things, which I thought I should share. These things are related to both the platforms (Windows vs. Linux), the tools and the architectural decisions.
I needed lots off data to test the solution I was contemplating, which involved XML processing, to see how constructing and parsing XML affects solution performance. To make it easier to compare timing differences I though I should use lots of records.
Occasionally one needs to pick up and process a large number of files, on the order of hundreds or thousands. With the Batch Inbound eWay/JCA Adapter it is not possible to pick up more then one file per poll. The Batch Local File, if triggered by some event other then an appearance of a file in a directory, perhaps a Scheduler trigger or a manual trigger, with correctly designed logic, can process many files in a single invocation.
It is required that each block of records starting with the H1 (header) record and containing all the following L1 (line) records, be written to a different file.
The solution involves the use of:
1. Batch Inbound eWay to locate the input file and provide its name and location to a Java Collaboration Definition
2. Batch Local File eWay to provide an Input Stream to the Batch Record eWay
3. Batch Record eWay to break up the input stream into records delimited by carriage return+new line
4. Batch Local File eWay to write each block of records to a file with a distinct name
Brief steps to implement this solution are given in the full Quick Note as QuickNote_001. The collaboration code will work in Java CAPS 5 and 6 Repository.