Your Mulesoft API needs to get data from some backend system that has no iterator for you to use and you can’t get the data in one go. For instance, some arbitrarily large set has to be retrieved but it is not possible to know how many elements exist before you retrieve them. This might prompt you to use recursion, or want to implement a while loop. However, Mulesoft has limited the amount of nested child contexts you may have per event in Mule 4. This article discusses a solution you might try.

An error message you might encounter looks a little like this,

Please refer to the Help Article for more details. You have exhausted the possibilities to solve it in a different way and need a simple solution to allow for loops with more iterations.

tl;dr: Mulesoft runtime has no way of imposing limits if you create a new event for each iteration, be sure to check that you don’t create infinite loops!

We want to fetch data from some source that doesn’t have an index or other API available that can help us structure calls. There is just some list, that is sorted and its too large to get all of it in one call. So we have to keep going back untill we got all of the items we’re interested in.

A pattern that you might want to use looks a little like the picture below. This effectively is a while loop. 

The problem is, that eventually the loop generates too many nested child contexts, which is not allowed by the runtime.

The only way to deal with this is not to generate nested child contexts but to create a new event. This article uses a VM, but you can substitute your own preferred transport barrier, as long as Mulesoft creates a new event with each iteration, you’re good.

Clone the repo and open it up in Anypoint studio. 

In the project you’ll find 2 large flows, get-all and get-limited and some supporting flows. Assumes some sorted collection.

This flow isn’t limited by the nested child limit, so it allows for infinite looping behavior. Note that the default behavior is to exit out of the loop. Because of the VM, you are creating a new event for each iteration. This is what allows the loop to go on forever.

It follows the pattern shown below.

It’s pretty simple. It collects a batch from the list, tests if the amount is smaller than expected, if not smaller go back for the next batch.

The minimal payload that arrives in the flow should have an offset and a batchSize, or some other way of keeping track of your progress. This is needed to iterate through the sorted collection and finish. It’s useful to have all metadata in the payload, as you will lose it after, publishing to the queue, as you’re creating a new event. Then you perform your actions, I recommend you aggregate only and do minimal processing within this flow. 

The test is simple, while the flow returns sizeOf(elements) == batchSize, we assume that there is more to get. If we get unlucky we have to do 1 extra operation that returns nothing. Otherwise the last operation will always have an element list smaller than the batchSize. For instance, the source has 100 elements, and we can get at most 2 per call. batchSize = 2 , this means we need to do 100/2 = 50 calls. sizeOf( [98, 99] ) = 2. This matches the condition, and the flow goes back for more, this time it will nog get more elements, so sizeOf( [] ) = 0. The flow doesn’t have more data to retrieve and exits. If the amount was 101, the last call would be sizeOf ( [ 100] ) = 1  and also exit.

Hopefully your system will allow for larger batch sizes than this example.

As you can see, it’s really important that the order of the elements doesn’t change when you retrieve it. Otherwise you are at risk for creating infinite loops.

is the implementation that allows for some recursion but is ultimately bound by the limit set by mulesoft. It follows the pattern, shown in the introduction. Note that the reference to the flow doesn’t have to be a synchronous call. For instance, wrapping a flow-ref with a async scope etc will keep the limitation. As long as Mulesoft creates an nested-child then you are bound by the limitation of 44. 

I have provided some basic tests to validate that the solution works. You can run the Munits, to validate that using the VM fixes the ‘issue’. You can compare the following tests:

When run, the test  completes and reports one call being detected. The console shows that the flow has been executed more than once. But as far as the runtime is concerned, only 1 event passed through. This is the result of creating new events for each iteration.

 won’t complete, and I’ve set the timeout to be 5000 mS, after which it will fail. If you set your console settings to allow a high number, you can scroll up and verify that the last log contains “offset: 43: Some result”. This is the limit imposed by Mulesoft.

If you want to play with the project to investigate how different offsets and batchSizes affect the outcomes. You can import the postman collection from the root of the project, or use the curl commands below. 

Generate different collection with: