Stephen W. Thomas BizTalk Blog

Debatching Options and Performance Considerations in BizTalk 2004

Download This Article and Sample Code Here: Debatching Options and Performance Considerations in BizTalk 2004 White Paper and Sample Code

Related Sample: Xml Envelope Debatching

In some business scenarios you may be required to receive a batch file that must be broken up and processed as single messages by BizTalk Server 2004.  This could allow for individual record level processing, selective content based routing, or single message mapping. 

General Debatching Design Considerations

Here are some general design considerations you might want to think about when planning a debatching approach with BizTalk Server 2004. 

General Debatching Design Considerations

• Header and Trailer Validation Requirements for Flat Files
• All or Nothing Debatching
  • Should single records be allowed to fail some type of validation, mapping, or routing
• Mapping Required
  • As a whole message
  • As single messages
• Last Message Indicator Required
  • Is another process depending on knowing when the last message finished
• Ordered Message Processing
• Time of Day Processing
  • Will this affect other processes running
• File Size (less important than in past versions of BizTalk)
• Record Volume

Although this article is focused on general debatching options, I want to go into more detail on some of the design considerations above.

Header and Trailer Validation

Typically batch files are received in a format other than XML, such as flat file.  In this case, the file will typically have a Header and Trailer record.  These records typically contain information about the number of records in the file, date, time, etc.  In some business processes this information needs to be validated prior to processing the file.  This creates some interesting design challenges.

Some options for this type of validation include a pre-debatching map, validation of the file using .net, or validation inside an Orchestration.  The best option depends on message size since some batch files can be very large (I consider very large as greater than half a gigabyte as XML).

Last Message Indicator Required

Debatching implies breaking up a file into many different pieces.  Sometimes, the single items must still behave as a batch.  Some business processes require knowing when the last message in the batch has been processed to activate another process.  In addition, sometimes ordered processing of the debatching messages is required. 

Ordered Message Processing

Ordered processing of the messages can be accomplished in a few ways.  One way is to use an ordered delivery supported adapter, like MSMQt.  This would require the debatcher to write the messages in the correct order to the queue for processing.  This may also require the use of a convoy to route all the single messages to the same business process. The challenge is to allow for ordered delivery without significantly affecting performance.

BizTalk 2004 Debatching Options

BizTalk 2004 provides us with several different methods for batch file debatching.  What is the best way to split up your files?  As always, that depends on your exact business scenario. 

In this posting, I will look at four different debatching options, review the performance data, and explain the benefits of each type.  I also have the test Orchestrations I used available for download.  I do not provide any sample files, but you can make your own since the Orchestrations use untyped messages.  Just make your structure like: DataData.

The four methods I will cover are:

• Receive Port Pipeline Debatching
• Orchestration XPath Debatching
• Orchestration Atomic Scope Node List Debatching
• Orchestration Outside .Net Component Debatching

Debatching Options Performance Test Results

Here are the results of the performance tests on each type of debatching.  Tests were run on a 2.4 GHz desktop with 1.25GB RAM.  The sample file produced single records that were 3 KB each.  No mapping was done on the files and times do not include time to send the files using a Send Port.  This is just the time to run the pipeline or orchestrations.  Throughput is intended to show a general idea of the amount of data running through the process; it is not the overall system throughput.  Additional tests were run for XPath and Node List that produced larger output files of 29.9 KB and 299.0 KB.