Task Factory

Available in Dataristix Core Service 2.0.150 and Dataristix Core App 2.0.243 and later, the Task Factory feature allows generating similar Tasks en masse with varying parameters, for single tasks or entire folders of tasks.

Use case - replicate single task

Production Line Task

Let’s assume we have a task “ProductionLine1” that stores counters for a production line. The task may receive data from an MQTT topic and stores data into an ODBC database whenever data changes. The task may look like this:

In this example, the MQTT device delivers a JSON object:

{
    "Counter1": 123,
    "Counter2": 123,
    "Counter3": 123
}

The JSON object is decoded by the MQTT client topic, with each individual tag defining a JSON path to the respective counter property within the object.

For “Counter1”, for example, tag settings define a JSON path “Counter1” to extract the “Counter1” value from the JSON object:

On the ODBC side, the topic for production line 1 contains three tags representing field names within a “ProductionLineCounters” database table to hold counters for one or more production lines.

The ODBC topic settings are as follows.

Observations

As you can see, the ODBC topic is specific to production line 1, because:

• The connector topic is named after production line 1
• The WHERE clause for table updates refers to production line 1

The MQTT topic is specific to production line 1, because:

• The connector topic is named after production line 1
• The tag name refers to the MQTT topic for production line 1
• The MQTT topic refers to production line 1

Finally, the task itself is named “ProductionLine1”.

If the production site were to add more production lines, then additional MQTT topics, ODBC topics and another task would need to be created to define additional dataflows, one for each additional production line.

Use of Task Factory

The Task Factory takes a task as a template, then substitutes text within the task and associated connector topics to create new tasks. In this instance, the “ProductionLine1” task will be the template, and whereever there is a reference specific to production line 1, it should be replaced by a new reference for production line 2, production line 3, and so forth.

Task factory parameters are lines of text, and the basic rules are as follows (more details can be found at the end of this chapter):

1. Columns within the first line define the text to replace (the text search is case-sensitive).
2. Columns in subsequent rows define new text that will replace the old text for new tasks.
3. For each row, excluding the first row, a new task will be created.

We therefore define the following factory parameters.

ProductionLine1	production/line1	ProductionLine=1
ProductionLine2	production/line2	ProductionLine=2
ProductionLine3	production/line3	ProductionLine=3

The header row defines the text that is to be replaced for new tasks. The text “ProductionLine1” is the original task name, the name of the MQTT “Subscribe” topic, and it is part of the ODBC topic name (“ProductionLine1Counters”). We do not need to define separate factory parameters for different connectors (but we could do so with “Scopes” as we will see later). The text “production/line1” appears in MQTT tag names and MQTT topics. The text “ProductionLine=1” appears in the WHERE clause of database queries defined for the ODBC topic. With these three substitutions, all “Production Line 1” specifics are covered.

Right-click on the “ProductionLine1” task and select the “Task Factory” menu item to open a dialog prompting to enter Task Factory parameters, either separated by comma, or separated by Tab characters. Preparing factory parameters in Excel and copying straight into the factory parameter dialog works fine. We should end up with the following.

Click on the “Next” button to preview changes that are to be applied. Check the “Verbose” option to get more detailed information. In this case, we see that two more tasks will be created as expected:

Click on the “Execute” button to create new tasks. The result will be:

New tasks will be placed into a new folder, in this instance into the “TaskFactory1” folder. Click “Finish” to view new tasks.

Task nodes within, for example, the “ProductionLine2” task are updated as expected, referring to new MQTT and ODBC topics specific to production line 2.

Additional MQTT topics are created for each production line, with MQTT topics and tag settings adjusted for the specific production line.

Finally, new ODBC topics are created that update the correct table row specific to each new production line.

Tip

To replace key-value pairs in Task Constants, define the text to replace in the form <constant name>:<constant value>. For example, to replace a Task Constant “UniqueIndex” that has a value of “1”, with a constant of the same name that has a value of “2”, define factory parameters as:

UniqueIndex:1
UniqueIndex:2

Use case - replicate multiple tasks

Replace in tasks

Select multiple tasks to search and replace text defined in factory parameters for each task. The effect is, that for each row (except of the first row) new tasks are created for each selected task that has substitutions. For example, select two tasks “AssemblyLine1” and “PackagingLine1” (or a folder containing them) and invoke the Task Factory with the following parameters:

Line1
Line2

The result will be that two new tasks are created, called “AssemblyLine2” and “PackagingLine2” (you will need to add other factory parameters columns as required for any connector topic and tag substitutions).

Replace in tasks and create providers

In another scenario, we may have a set of tasks that, together, form the data collection for a site. Let’s assume that site “Wellington” has an assembly line and a packaging line. The MQTT and ODBC configuration may look like this:

Note that there is a specific MQTT client and a specific factory database for “Wellington” as the providers of source and destination connector topics.

Tasks for site “Wellington” are as follows.

We now want to replicate the setup for site “London” and “Paris”. The factory parameters are:

Wellington
London
Paris

This means, that the Task Factory will replace all occurences of “Wellington” with either “London” or “Paris” in new tasks. Because there is a dedicated MQTT client “MQTT Wellington” and dedicated database “FactoryDb Wellington”, a new MQTT client and database entry will also be created, however, credentials will need to be configured separately.

Note that you may need to add columns for connector topic or tag substitutions, but here we assume that this is all that is needed. Note also that text search is case-sensitive, so if there was text “wellington” (lower case) that needed to be substituted, then another column for “wellington” should be added.

When invoking the Task Factory on a folder (by selecting the “Task Factory” menu item in the folder context menu), then the folder name itself is also subject to substitutions:

In this case the Task Factory results are:

The MQTT and ODBC connectors have additional MQTT clients, databases, and topics added:

Tip

If a single set of Task Factory parameters cannot achieve the desired outcome, consider running the Task Factory more than once with different sets of parameters.

Reference

Scopes

Scopes in factory parameters restrict text substitutions to specific tasks, connectors, connector topics, or connector tags. If, for example, there is some text that appears in connector topic setting and also in connector tag settings, but only the text in a connector tag should be replaced, then a Scope may be defined.

Scope definitions must be defined in columns of the first row of the factory parameters. If the first row defines a scope in any column, then all other columns in that row must either be empty or also define a scope. Connector names used in scopes should match the connector names that are shown in the navigation tree. Scopes take on one of the following forms:

1. Scope:Template

   Restricts text search of an entire row of factory parameters to a specific task in multi-task replications. The column value should be the name of the task to be used as the template for this row.

2. Scope:Core

   Restricts text search and substitutions for this column to Core tasks, folders, and task nodes.

3. Scope:Core Task

   Restricts text search and substitutions for this column to Core tasks.

4. Scope:Core Folder

   Restricts text search and substitutions for this column to Core folders.

5. Scope:Core Task Node

   Restricts text search and substitutions for this column to Core task nodes.

6. Scope:Core Task Node Constant

   Restricts text search and substitutions for this column to Core task constants. Columns should define text to replace in the form <constant name>:<constant value>.

7. Scope:<connector name>

   Restrict text search and substitutions for this column to the named connector.

8. Scope:<connector name> Topic

   Restrict text search and substitutions for this column to topics of the named connector.

9. Scope:<connector name> Tag

   Restrict text search and substitutions for this column to tags of the named connector.

For example, the following factory parameters restrict text search and substitutions for text “production/line1” to MQTT tags.

	Scope: MQTT Tag
ProductionLine1	production/line1	ProductionLine=1
ProductionLine2	production/line2	ProductionLine=2
ProductionLine3	production/line3	ProductionLine=3

Supported Targets

The following substitutions are supported in tasks, connector topics, and connector tags.

Core/Connector	Substitution Target
Core	Folder Name, Task Name, Task Node Name, Task Constant Name and Value
CSV	Topic Name
E-Mail	Topic Name, Subject, Recipients, Message Body
Excel	Topic Name
Google Sheets	Topic Name, Tag Name
IoT Devices	(No substitutions)
Kafka	Client Name, Client ID, Topic Consumer Group ID, Consumer Commit Topic, Producer Partition Key, Tag Name, Kafka Topic, JSON Path
MQTT	Client Name, Client ID, Topic Name, Tag Name, Tag MQTT Topic, Tag JSON path
MySQL	Database Name, Auto-Table Prefix, Topic Name, Table Name, Custom Queries, WHERE clause
ODBC (32-bit)	Database Name, Auto-Table Prefix, Topic Name, Table Name, Custom Queries, WHERE clause
ODBC	Database Name, Auto-Table Prefix, Topic Name, Table Name, Custom Queries, WHERE clause
OPC DA	OPC Server Name, Topic Name, Tag Name, Tag Item ID
OPC UA	OPC Server Name, Topic Name, Tag Name, Tag Node ID
Oracle	Database Name, Auto-Table Prefix, Topic Name, Table Name, Custom Queries, WHERE clause
PostgreSQL	Database Name, Auto-Table Prefix, Topic Name, Table Name, Custom Queries, WHERE clause
Power BI	Topic Name
REST	Client Name, Topic Name, Tag Name, Relative URL, JSON Path to Tag Objects, JSON Tag Object Property containing the tag name, JSON Tag Object Name identifying the tag, JSON Tag Object Property containing the tag value
Script	Topic Name, Script, Tag Name
SOAP	(Not substitutions)
SQL Server	Database Name, Auto-Table Prefix, Topic Name, Table Name, Custom Queries, WHERE clause
SQLite	Database Name, Auto-Table Prefix, Topic Name, Table Name, Custom Queries, WHERE clause