Sunday, May 8, 2016

Multi-threaded batch job template

Batch jobs in AX are important concepts, required for processing different business logic, filling up tables for reports etc

Multi-threaded batch jobs are an advanced usage of the batch jobs functionality, which can handle even more heavy duty tasks due to parallel processing capability.

Since these jobs have been becoming common place, I thought to create a template to make my life easier.

It’s a project with few objects.

Few pointers:
  1. Objects have prefix Template, which I replace with the feature I want the batch job for.
  2. "TemplateService" class, "createTasks" method is where the magic happens. This methods spins off a fixed number of threads depending on server thread setting, usually 8. It divides the workload in these 8 threads and makes a list of SysOperationServiceController objects. These objects are basically call to the “run” method of the class, which has the main business logic. For example, you want to process records in CustTable. If you have 32 customers, you will get 8 threads working on a set of (32/8) 4 customers each. This is not the best usage scenario for multi-threading but think 32000 customers, meaning each thread handles 4000 customers in parallel. Your job would finish in roughly 1/8th the time required to finish a single threaded batch job. A less common way to do multi-threading, would be to spin as many threads as required. For example if we want to iterate through all the customers and do some logic for each, we could spin 32 threads. Will try to modify this template to suit that requirement.
  3. Another nice logic is in method “AddrunTimeTaskList” in class “ANSysOperationServiceBase”. This class extends SysOperationServiceBase, to enable adding dependent task to a list of parallel tasks. Please see my previous blog here for more info on this. My template project shows how to use this class and add a dependent method to a list of tasks.
  4. There are “runInitialTask” and “runFinalTask” methods which can be used to call tasks before and after your parallel list of tasks. 
  5. I use paging in query to divide the workload. This requires your AOT query to have its order by node set. 
Please email me for xpo. Cheers.

1 comment:

  1. Hi Aman,
    I was looking exactly the same logic for my customization. Actually I have a table with thousands of records and have to create multithreading so that I can divide the records among those threads lets say I have 100 records and 5 threads. So need to divide 20 records per thread. I am not getting that logic using SYSOperation framework multithreading. Please send me the XPO on