Speed up filtering large arrays in returned dataset

Magic Suite ReportMagic
1

In a recent project to build a custom Jira report, a large set of data with many custom fields are being returned, although only a small number of these fields are required for the report itself.
The original approach to filtering this data was to use a ForEach macro with a Switch macro to select only the custom fields of interest:

// Original code, using a Foreach and Switch across the full jArray of CustomFields to find fields of interest

[.: JiraIssue.CustomFields, =>CustomFields]
[ForEach: values=`{=CustomFields}`, =>CustomField]
     [.: CustomField.Name, =>Name]
     [Switch: value='{Name}']
         [Case: value='Field_Of_Interest_01']
             [.: CustomField.Values, =>Values]
             // Do interesting stuff with 'Values' object for Field_Of_Interest_01
 
         [Case: value='Field_Of_Interest_02']
             [.: CustomField.Values, =>Values]
             // Do different interesting stuff with 'Values' object for Field_Of_Interest_02
 
         [Case: value='Field_Of_Interest_03']
             [.: CustomField.Values, =>Values]
             // Do other interesting stuff with 'Values' object for Field_Of_Interest_03
 
         [Default:]
     [EndSwitch:]
 [EndForEach:]

While functionally correct, on the full production data set, this could run for several hours!
In an effort to optimize this code, the NCalc extensions ‘where()’ function was used to filter the list for each field of interest as below:

// Optimized code, using NCalc where and or List.Where macro to find fields of interest
[.: JiraIssue.CustomFields, =>CustomFields]
[=: `where( CustomFields, 'f', 'jPath(f,\'Name\') == \'Field_Of_Interest_01\'')`, =>CustomField]
[=: `jPath(itemAtIndex(CustomField,0),'Values')`, =>Values]
// Do interesting stuff with 'Values' object for Field_Of_Interest_01

[=: `where( CustomFields, 'f', 'jPath(f,\'Name\') == \'Field_Of_Interest_02\'')`, =>CustomField]
[=: `jPath(itemAtIndex(CustomField,0),'Values')`, =>Values]
// Do different interesting stuff with 'Values' object for Field_Of_Interest_02
 
[=: `where( CustomFields, 'f', 'jPath(f,\'Name\') == \'Field_Of_Interest_03\'')`, =>CustomField]
[=: `jPath(itemAtIndex(CustomField,0),'Values')`, =>Values]
// Do other interesting stuff with 'Values' object for Field_Of_Interest_03

For the same end result, the run-time on the full production data set reduced to a small number of minutes!
One small point to note is that:
• In the original Foreach code, CustomField is a jObject
• In the NCalc where() code, CustomField is actually an jArray containing a single jObject. Therefore, the itemAtIndex() function should used to retrieve the Values object

1 Like