June 2022


Prev		Next

This month has seen a lot of exciting developments that will be of interest to organizations that have complex and changing models, or who wish to build model templates that can be reused for project that are similar but have slightly different inputs.

The Patchworks system can now load data tables directly from Microsoft Excel 'XSLS' and 'XLXB' types of workbooks. When loading a table the workbook is opened in read-only mode, the cells are copied in to a memory-based table, and the workbook is closed. The table has the same interface as all other Patchworks tables, and can be used in the same ways as other tables: for report, as a lookup table, etc. See the ExcelStore documentation for further details.
Organizations who have other data loading needs should contact SPS to discuss additional options.
The curvetable and table ForestModel elements can now load yield curve and lookup table data directly from tables, including from any supported table format. Curve data is supported in both row and column oriented formats, and reselection expressions can be used to limit the data that is loaded. All of the same features for label extraction, curve scaling and shaping are available with table input.
ForestModel has a new generalized repeat element that can be used to enclose almost all other elements. Repeat elements provide a powerful and convenient way to build models, particularly when faced with similar declarations that only vary in a few details. The short-lived <repeatselect> and <repeattreatment> elements have been deprecated and removed now that there is a uniform method to declare repeat blocks almost anywhere.
Repeat blocks may be nested to arbitrary depths and interleaved with other ForestModel elements. Each repeat element requires a name that is unique within the elements that is enclosed within.
The repeating values within these blocks come from three possible source:
- counting loops using from, to and by parameters. The named loop variable contains the index value.
- list based loops that stream through the elements in a provided list. The named loop variable contains the list item.
- table-based loops that iterate through the selected records from a lookup table. The named list item contains the row number, and additional named repeat variables are created holding the values for each cell in the row.
Regardless of the value source the repeat blocks perform uniformly. Enclosed elements are processed at each iteration, and all enclosed Patchworks Query Language expressions can access the named repeat values using the getEnv function.
Implementing the new repeat blocks required moving to a new schema validation system. Previously ForestModel was validated by the ForestModel.dtd Document Type Definition, and the new grammar uses the ForestModel.xsd XML Schema description. The MatrixBuilder will continue to correctly process all ForestModels based on the DTD definition, however new features cannot be back ported, and will only be available with the new system. The most recent Simple ForestModel Generator Spreadsheet has been upgraded to make use of the new schema and the new repeat block features. Contact SPS support if you have any difficulties upgrading to the new version.
Lists are now first class syntactic objects in the Patchworks Query Language. This new syntax is simpler and makes it easier to write and review list expression.
Previously lists were created using the list or split functions, or were automatically split from strings by some of the list-aware functions (e.g. lookuptable and others). The new syntax uses brace character to demarcate the list value:
```
{1, 2, 3, 4, 5}
{'a', 'b', 'c'}
{{'a', 'b', 'c'},{'d','e', 'f'}}
```
List elements can be any Patchworks Query Language expression, so long as all values within the list have the same semantic data type (e.g. all strings, or all booleans, all numbers or all lists). List elements may also be other lists (very useful in some situations), nested to an arbitrary depth, so long as all lists contain the same data type.
The Patchworks Query Language has a number of new functions to support list operations.
- append appends lists and values to form new lists.
- curvenames introspects on the loaded set of curves and returns a list of curve labels.
- filter selects values from lists using pattern matching, and provides for optional pattern extraction.
- join provides a complement to the split function, and concatenates list values in to a string.
- reverse reverses the order of elements in a list.
- sort sorts list elements in to their natural order.
- tablelist extracts values from a lookup table in to a list using Patchworks Query Language expressions to control the value calculation and optionally the selected records.
- unique reduces the items in a list to unique values, discarding duplicates.
A new eval function provides a mechanism to compile Patchworks Query Language expressions that have been composed used any of the string, list or data access operators. This can, for example, be used with the curvenames or tablelist operators to introspect on the input dataset and dynamically generate treatment or attributes that match the input data sets. Consider the following example for an idea of what can be done:
```
eval(join(sort(unique(filter(curveNames(),'.*\.Y(\w+)',
'attribute(\'%f.Yield.{1}\')'))),'+'))
```
Trying to comprehend a complex expression can be daunting, so let's reformat the above line to break the complexity down in to easier to understand component pieces:
```
 eval(
   join(
     sort(
       unique(
         filter(
           curveNames(),
           '.*\.Y(\w+)',
           'attribute(\'%f.Yield.{1}\')'
         )
       )
     ),
     '+'
   )
 )
```
To follow the flow of data in this expression we work from the innermost back to the outside:
1. Use the curvenames function to retrieve the names of all of the curves in the curve library
2. filter the curve name list with the following parameters:
  - '.*\.Y(\w+)' is a regular expression that will select all curve names that end with '.Y' and other alphabetic characters. The trailing alphabetic are captured in match group 1.
  - 'attribute(\'%f.Yield.{1}\')' is a translation template that interpolates the contents of match group 1 in to the attribute function. The single quotes are escaped because they are already contained within a quoted string.
3. The unique function reduces the list by discarding duplicates.
4. sort the list
5. join the list elements together interspersing the list values with the + symbol, producing a string value. At this point the resulting string will look similar to
```
attribute('%f.Yield.ce')+attribute('%f.Yield.df')+attribute('%f.Yield.pn') ... etc
```
6. Use the eval function to compile the string contents in to code that is ready for execution.
At this point the expression is ready to use to attach attributes to blocks using the standard ForestModel elements. No knowledge of species codes was required when the ForestModel was written. All attribute names and values can be determined from the inputs as they are presented, allowing great opportunities to develop fully generalized template models that correctly adapt to a variety of different input datasets.
The are a large number of different ways that the new set of functions and ForestModel elements can be composed together to make model 'templates', and like most technologies it will depend on each individual project to determine the role for these options.
A new exec element has been added to the ForestModel language. This element allows the specification of scripts that can be run using the Powershell, Cmd or BeanShell command processors. The intent of this feature is to allow the MatrixBuilder to ensure that dependencies are met before the input matrix is built. Actions could include downloading or preprocessing required files.

In other news,

The Block Attribute table provides a window in to the current state of the simulation. In addition to displaying information, some of the columns in this table accept edits that will change the state of the simulation, such as the current treatment and the exclusive account choices. Previously these fields accepted text edits, where the new value would need to be typed in to the field. With this version these editable columns provide a pop-up menu that lists the eligible options. Pointing and clicking to select a value from the menu will make a change to the internal model state. This method is easier and more reliable than data entry, because only acceptable values are shown in the popup menu. All of the programmatic methods for updating these columns continue to work in the same way.