Just now I had an occasion to work with an integration solution intended to process lots of records. By lots I mean over 1 million smallish records. My customary platform to experiment on is Windows XP. Lots of reasons for that, most of them historical – I have tools I know and like and so on. While trying to work with such a volume of data I noticed a number of “interesting” things, which I thought I should share. These things are related to both the platforms (Windows vs. Linux), the tools and the architectural decisions.
I needed lots off data to test the solution I was contemplating, which involved XML processing, to see how constructing and parsing XML affects solution performance. To make it easier to compare timing differences I though I should use lots of records.
The discoveries are discussed in Right tools for the job.pdf.
