Topics¶
Terminology¶
Dataristix uses the term “Topic” as a term for “Tag Containers”. Connector topics and tags are connected in Tasks to define data flows.
Apache Kafka uses the term “Topic” as an “address” where application messages are stored or retrieved. Producer clients send payloads with a Kafka topic “address” to the Kafka brokers and consuming clients receive these payloads when subscribing to such Kafka topics.
In the context of the Connector for Kafka, “Connector Topics” are configured for each Kafka client either as “Producer” or “Consumer” topics and refer to the topics in the Dataristix sense; “Tags” contained in a Producer or Consumer “Connector Topic” represent Kafka topics. Kafka topics (tags) within a Connector Topic share the same conector topic settings.
Add Topic¶
To send data to a Kafka broker, add a Producer connector topic by selecting the “Produce” navigation node for a Kafka client, then press the “Add” toolbar button. To receive data from a Kafka broker, add a consumer connector topic by selecting the “Consume” navigation node instead. This will add a new connector topic with default settings. Rename the topic to suit and adapt settings in the property panel.
“Consume” Topic Settings¶
Configure the following topic settings.
- Name
- Any connector topic name
- Consumer Group ID
- The consumer group used for message retrieval. The default consumer group ID consists of the Kafka client name and the topic name. Untick “Use default” to specifiy a different ID.
- Begin message processing with
- Determine how message consumption begins when the topic receives messages for the first time, when there is no commited offset stored for the consumer group. “Earliest message” means, the “Consume” topic will ask the Kafka broker to send all available messages. “Latest message” means, that the broker will only send new messages. This setting is sometimes referred to as Auto Offset Reset. Note that live values are not displayed for the connector topic as it may interfere with message delivery in tasks.
- Use external commit topic
- By default, consumer clients will commit received messages (mark them as “received” on the broker side) periodically. In the meantime, the Dataristix Core module should be able to deliver any messages to the final destination within task pipelines. For most cases nothing else needs to be done. However, if “at least once” delivery must be guaranteed, then an additional Kafka topic may be used as an “external commit topic” to inform the consumer when it can safely commit messages. This addresses the situation where the final destination is unable to store new messages, with the consumer committing messages before they reach the target. If a task is restarted, then the consumer may not receive these messages again, because they are already marked as “received” on the broker side. For “at least once” delivery, a message to the “external commit topic” should be sent when it is confirmed that the final destination has successfully stored a consumed message. Check option “Use external commit topic” to specify which topic will receive delivery confirmation in form of Kafka topic names, partitions and offsets. Please see the example at the end of this chapter.
“Produce” Topic Settings¶
Configure the following topic settings.
- Name
Any connector topic name
- Make this a generic topic
If checked then the connector topic is not expected to contain any tags and Tasks may send any tag to this topic for processing. Kafka topic names that are sent to the broker are generated automatically based on the connector topic name, the connected topic name, and the connected tag name within tasks.
When this option is checked then the following settings are also available:
- Normalize Kafka topic Identifiers
- Kafka topic names are converted to lower case characters and underscore characters.
Note that the broker must allow auto-generating topics.
- Partition Key (optional)
Optionally specify a partition key for messages sent by this producer topic.
Remove Topics¶
Remove topics by selecting the Kafka client in the navigation panel, then select one or more topics to remove in the content panel, and finally click on the “Remove” toolbar button to remove selected topics. You can also right-click on a topic and select “Remove” from the context menu.
Warning
Note that topics are also removed from any task using the topics.