First-In-first-Out (FIFO) or Ordered Processing from Mailbox

Some times there is a need to Process data arriving from partners or systems in the same order as it arrived in, or based on a specific order. There is different ways of doing such a thing.

In this post I will show two examples on processing files from a Mailbox. One is to process the files in the same order as the files arrived to the mailbox in (FIFO). The second example is to process the files based on the file name (the processing order is set by the sending system/partner). I will use the same Business Process, but with one parameter set different.

Mailbox

I start by creating some files. I created 6 simple Text Files. I gave them names including a number from 001 to 006. This number is used for the second example, where this number will indicate the order the files will be processed in.

Then I upload these 6 files to my Mailbox. I upload them in the opposite order (first 006, then 005 etc.) just to be able to see the difference in the processing of the two examples. So the result on the FIFO example will be to process the 006 file first, then the 005 etc. since that was the upload order.

My Mailbox will then look like below. The file names is numbered, and the ID (order uploaded) is the opposite.

Mailbox

Business Process

I will create a really simple Business Process just to show the point. The Processing in this example is just to write the file to a File System.

I start by creating an instance of a File System Adapter Service to use in my Business Process. I set the collection folder (it won’t be used, but is mandatory in the service configuration) to /SI_Cluster_Share/DatafolderTest/get and the Extraction Folder to /SI_Cluster_Share/DatafolderTest/put. This is where my files from the Mailbox will be put.

FSA_Config

Then I build a Business Process to find files on a specific Mailbox, and pick them in a specific Order.

I start by using the Mailbox Query Service to find Extractable messages with any filename on the Mailbox /esmines and order them based on MessageId. Then I create a variable named MsgCounter that is initially set to the value 0. I create a loop, that loops as long as MsgCounter is less than the number of messages in the Mailbox. For each iteration of the loop, I increase MsgCounter by one, selects a message by using the MsgCounter as an Index and writes the file to the File System. I give the files a file name consisting of the MsgCounter variable, the MessageId and the original MessageName to be able to see the data used and the order the files is processed in.

My Business Process graphically looks like below.

BP_GPM

FIFO Example

The Code for the FIFO example looks like below:

<process name="default">
  <rule name="MessagesLeft">
    <condition>count(/ProcessData/Message) > number(/ProcessData/MsgCounter)</condition>
  </rule>

  <sequence name="StartEvaluatingMbx">
    <operation name="Mailbox Query Service">
      <participant name="MailboxQuery"/>
      <output message="MailboxQueryServiceTypeInputMessage">
        <assign to="." from="*"></assign>
        <assign to="MailboxPath">/esmines</assign>
        <assign to="MessageExtractable">Yes</assign>
        <assign to="MessageNamePattern">*</assign>
        <assign to="OrderBy">MessageId</assign>
      </output>
      <input message="inmsg">
        <assign to="." from="*"></assign>
      </input>
    </operation>

    <assign name="Assign" to="MsgCounter">0</assign>
    <choice name="MessagesLeft">
      <select>
        <case ref="MessagesLeft" activity="PickMessage"/>
      </select>

      <sequence name="PickMessage">
        <assign name="Assign" to="MsgCounter" from="MsgCounter+1"></assign>
        <operation name="Mailbox Extract Begin Service">
          <participant name="MailboxExtractBegin"/>
          <output message="MailboxExtractBeginServiceTypeInputMessage">
            <assign to="." from="*"></assign>
            <assign to="CommitNow">Yes</assign>
            <assign to="MessageId" from="number(/ProcessData/Message[number(/ProcessData/MsgCounter)]/MessageId)"></assign>
          </output>
          <input message="inmsg">
            <assign to="." from="*"></assign>
          </input>
        </operation>

        <operation name="File System Adapter - ES_DatafolderTest Put">
          <participant name="ES_DatafolderTest"/>
          <output message="FileSystemInputMessage">
            <assign to="." from="*"></assign>
            <assign to="Action">FS_EXTRACT</assign>
            <assign to="assignedFilename" from="concat(/ProcessData/MsgCounter,'_',/ProcessData/MessageId,'_',/ProcessData/MessageName)"></assign>
            <assign to="assignFilename">true</assign>
          </output>
          <input message="inmsg">
            <assign to="." from="*"></assign>
          </input>
        </operation>

        <repeat name="Repeat" ref="MessagesLeft"/>

      </sequence>
    </choice>
  </sequence>
</process>

 

Test FIFO

To test this I upload the Business Process to my Sterling B2B Integrator Environment and run it.

The result in my File System looks like below. The File Name consists of the MsgCounter variable, which pretty much tells the order of the processing. Then it is the MessageId from the Mailbox system, which tells the order the files was uploaded in. And last, the Original File Name. So this shows that the Files was downloaded in the same order as they was uploaded.

FIFO_Result

File Name Example

To process files based on the File Name, I use the same Business Process, but change one parameter in the Mailbox Query Service. As shown in the Code below from the Business Process, I have changed the OrderBy variable. The rest of the Business Process stays the same as previous.

    <operation name="Mailbox Query Service">
      <participant name="MailboxQuery"/>
      <output message="MailboxQueryServiceTypeInputMessage">
        <assign to="." from="*"></assign>
        <assign to="MailboxPath">/esmines</assign>
        <assign to="MessageExtractable">Yes</assign>
        <assign to="MessageNamePattern">*</assign>
        <assign to="OrderBy">MessageName</assign>
      </output>
      <input message="inmsg">
        <assign to="." from="*"></assign>
      </input>
    </operation>

 

Test Message Name Processing

To test this, I do exactly the same as in the last test. I upload the edited version of the Business Process and runs it.

The result looks like below. Now the MsgCounter value is in the same order as the File Name order. Thus it is processed based on File Name.

FileName_Result

Summary

This is two simple examples on how to process Messages from a Mailbox in a specific order when that is needed. This does however not take in consideration anything around sequencing and missing files in a sequence. That is another story…

0 comments on “First-In-first-Out (FIFO) or Ordered Processing from MailboxAdd yours →

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: