Features

Data flow pipelines

A new type of pipeline, “Flow” pipelines, has been introduced in version 4.29. Flow pipelines ochestrate multi-step data flow by launching one or more “normal” or other flow pipelines. Batches created by a flow are grouped together as a single flow batch with an aggregate status, making it easier to monitor the overall success of the flow. It is also possible to call web services to interact with external applications, for example, to let other apps know that new data is ready in xMart. A flow can be configured to run in parallel (default) or sequence.

Example of a flow pipeline:

<XmartPipeline>
  <Flow Mode="SEQUENTIAL">
    <RunPipeline OriginCode="ABC" />
    <RunPipeline OriginCode="DEF" />
    <CallWebService Url="http://webservice1/get" />
    <Flow Mode="PARALLEL">
      <RunPipeline OriginCode="GHI" />
      <RunPipeline OriginCode="JKL" />
      <CallWebService Url="http://webservice2/get" />
    </Flow>
  </RunPipeline>
</Flow>
</XmartPipeline>

The PostRun section of “normal” pipelines is also a flow. So you have the option of using PostRun in a normal pipeline or replacing that by a new pipeline of type Flow.

flow in latest 50 batches

For more information, read the article on data flow pipelines

Enhancements

Pipeline documentation

The pipeline documentation has been re-organized and updated and a new introduction to pipelines article is available.

Date filter tree

Clicking on the header of any date or date-time column now displays unique date values as a tree, so it’s easier to assess distinct date values by Year, Month, and Day.

data filter tree

Column resizing improved

The frustration users experience when resizing columns on the data view page has been remedied by replacing the underlying gridview component on the data view, data view history, forms list page and form data pages with the free open source library Tabulator.

Columns should resize more easily and reliably now. It is also possible to double click any vertical column separator to auto-size the column to fit the data.

New table view for Forms list page

The list of forms can be viewed as a table to compliment the existing cards view. It should be easier to see the details in this view and to sort the forms.

forms list table view

Reduce Catalog tag list when a tag is selected

In the data catalog, when a tag is selected, the tags also attached to the same objects are emphasized, making it easier to locate objects that share the same tags.

reduce tag list

Display label for pipeline variables

A new, optional “Label” property on pipeline variables allows customizing the text when prompting the user, rather than showing the name of the variable.

variable label

Remove user from mart or role on users page

You can now remove a user from a mart or roles on the user list page, instead of having to open each role and remove the user. Once the user is selected, the buttons to remove the user will appear.

remove user from mart or roles

Pipeline and custom SQL view editor improvements

The underlying component of the pipeline and custom view editors has been changed to the Monaco editor. This brings some new features and hopefully usability improvements to pipeline and sql view editors.

  • You can change the visual theme of the editor now: light, dark, high contrast, low contrast.
  • Mini map to simplify navigation of lengthy code

pipeline editor improvements

Both the pipeline and the custom sql view editors have a settings button to customize the editor.

pipeline and custom sql view editor settings

Slow custom view management

The ability for users to create custom SQL views is a very useful feature but, for several reasons, they can be very, very slow and under the right conditions bring down the performance of the entire system. A number of changes have been applied to manage this situation.

  • The speed of a custom view is now measured and can be categorized as slow.
  • Slow custom views may or may not succeed in having their rows counted, so not all objects will have row counts in the data catalog (but all tables will have row counts).
  • On the data view page, sorting and paging of slow views has been disabled. Filtering is possible and it is possible to “load” more rows into the data view and also change the page size but you can’t advance to a particular page number.

  • An internal change was applied to minimize the number of times an object has its rows counted.

    Other changes

  • #5170: Load page slowness - batch short history takes > 1min
  • #5451: SQL Editor User Feedback
  • #5510: Websockets not connecting with Firefox on Stage / UAT
  • #5491: Better message to user when initiating a rollback
  • #5497: Date-related transforms may/do not work with strict data types
  • #5458: Ability to un-assign role directly and un-assign user directly from user list page
  • #5488: Duplicate long-running row counting queries
  • #5507: HierarchyLookup - value starting with a space not properly matched with synonym
  • #5519: LookupInterval issue in EMRO_VPI/data/REF_PLACES for rows where ONSET_DATE = EffEndDate
  • #5524: Mart deploy: catalog views tags and categories are not imported
  • #5418: Custom view editor - add auto-complete
  • #5540: Batch error if MartLookup Tag is too long
  • #5535: Empty (deleted?) custom view exported in mart package fails at import
  • #5571: GetText generates too many errors when parsing wrong XLS file
  • #5489: Dataflow UI tweaks
  • #5492: Why is the MART_ID changing to the title all of a sudden?
  • #5166: Clean up scheduled jobs (disable old jobs)
  • #5466: WebApp deployment - avoid locking issues
  • #5534: Form BOOLEAN in DEFAULT_VALUE and in HIDE_EXPRESSION not working as excepted
  • #5508: Make the xmart script-guide public and LLM-friendly