May 08

Right tools for the job? – Interesting discoveries when processing large volume of data

By Michael Czapski Sun No Comments »

Just now I had an occasion to work with an integration solution intended to process lots of records. By lots I mean over 1 million smallish records. My customary platform to experiment on is Windows XP. Lots of reasons for that, most of them historical – I have tools I know and like and so on. While trying to work with such a volume of data I noticed a number of “interesting” things, which I thought I should share. These things are related to both the platforms (Windows vs. Linux), the tools and the architectural decisions.

I needed lots off data to test the solution I was contemplating, which involved XML processing, to see how constructing and parsing XML affects solution performance. To make it easier to compare timing differences I though I should use lots of records.

The discoveries are discussed in Right tools for the job.pdf.

Tagged with: • • • • • •
May 07

Getting Hundreds of Files using Batch Local File eWay in Java CAPS 6

By Michael Czapski Sun 4 Comments »

Occasionally one needs to pick up and process a large number of files, on the order of hundreds or thousands. With the Batch Inbound eWay/JCA Adapter it is not possible to pick up more then one file per poll. The Batch Local File, if triggered by some event other then an appearance of a file in a directory, perhaps a Scheduler trigger or a manual trigger, with correctly designed logic, can process many files in a single invocation.

The document, ProcessingHundredsOfFileWithBatchAdapter.pdf, discusses how Batch Local File-based solution can be constructed to effectively process hundreds of files in a single pass.

This article references a ZIP archive “ProcessingHundredsOfFileWithBatchAdapter.zip“.

Tagged with: • • • • • • • •
preload preload preload