May 08

Right tools for the job? – Interesting discoveries when processing large volume of data

By Michael Czapski Sun No Comments »

Just now I had an occasion to work with an integration solution intended to process lots of records. By lots I mean over 1 million smallish records. My customary platform to experiment on is Windows XP. Lots of reasons for that, most of them historical – I have tools I know and like and so on. While trying to work with such a volume of data I noticed a number of “interesting” things, which I thought I should share. These things are related to both the platforms (Windows vs. Linux), the tools and the architectural decisions.

I needed lots off data to test the solution I was contemplating, which involved XML processing, to see how constructing and parsing XML affects solution performance. To make it easier to compare timing differences I though I should use lots of records.

The discoveries are discussed in Right tools for the job.pdf.

Tagged with: • • • • • •
May 07

Getting Hundreds of Files using Batch Local File eWay in Java CAPS 6

By Michael Czapski Sun 4 Comments »

Occasionally one needs to pick up and process a large number of files, on the order of hundreds or thousands. With the Batch Inbound eWay/JCA Adapter it is not possible to pick up more then one file per poll. The Batch Local File, if triggered by some event other then an appearance of a file in a directory, perhaps a Scheduler trigger or a manual trigger, with correctly designed logic, can process many files in a single invocation.

The document, ProcessingHundredsOfFileWithBatchAdapter.pdf, discusses how Batch Local File-based solution can be constructed to effectively process hundreds of files in a single pass.

This article references a ZIP archive “ProcessingHundredsOfFileWithBatchAdapter.zip“.

Tagged with: • • • • • • • •
Feb 27

Java CAPS Quick Note 001 – Split file into multiple files

By Michael Czapski Sun 2 Comments »

This Quick Note discusses a solution to the use case provided by Richard Kuiter.

An input file contains the following records:

H100000000000014099120ASN00507
L1140991200000008261850826185738
L1140991200000008261850826185738
L1140991200000008261850826185738
L1140991200000008261850826185738
L1140991200000008261850826185738
L1140991200000008261850826185738
H100000000000014099126ASN00531
L1140991260000008262690826269662
L1140991260000008262690826269662
L1140991260000008262690826269662
L1140991260000008262690826269662
L1140991260000008262690826269662

It is required that each block of records starting with the H1 (header) record and containing all the following L1 (line) records, be written to a different file.

The solution involves the use of:
1.    Batch Inbound eWay to locate the input file and provide its name and location to a Java Collaboration Definition
2.    Batch Local File eWay to provide an Input Stream to the Batch Record eWay
3.    Batch Record eWay to break up the input stream into records delimited by carriage return+new line
4.    Batch Local File eWay to write each block of records to a file with a distinct name

Brief steps to implement this solution are given in the full Quick Note as QuickNote_001. The collaboration code will work in Java CAPS 5 and 6 Repository.

Tagged with: • • • • • • • • • • • • • • •
preload preload preload