Handling very large messages in a messaging solution may require memory resources many times greater than the size of the largest message to be handled. Frequently the architect has no choice but to consume or produce a very large message, a file containing a batch or related transactions, for example, or a large and complex XML message generated by, or intended for, an external application. Handling such messages poses special challenges.
Java CAPS can assist with Batch eWay support for data streaming when large messages are manifested as files in a file system.
If it is possible to break large messages up into components and process components individually, or collect components and assemble them into a large message. eTL, another of the products in the Java CAPS Suite, can assist in processing large volumes of data. Whilst ETL (Extract, Transfer and Load) is typically associated with one off batch extraction and load of data, Java CAPS’ eTL can be used both standalone and in-stream as part of a larger Java CAPS solution. In this in-stream mode it will be discussed as a possible means of streaming data between a flat file and database table or between
database tables/views.
This extract from the soon–to-be-released book, “Java CAPS Basics – Implementing Common EAI Patterns”, discusses the Java CAPS 5.1 Batch eWay streaming facilities, and presents and compares a number of data streaming implementations, including an eTL implementation.
Link to Java CAPS 5.1.3 Data Streaming section of the manuscript.
