Kafka Clients

The connector for Apache Kafka functions as one or more Kafka client(s). Each client may have different configuration settings.

Add Kafka Client

To add a client, select the “Clients” navigation node and press the “Add” toolbar button. This will add a new Kafka client with default configuration settings. Rename the client to suit.

../_images/kafka-add-client.png

Client Settings

After selecting the client’s “Configuration” node, edit the following settings as required.

../_images/kafka-configure-client.png
  • Name
    Choose any name.
  • Client ID
    Configure the Kafka client ID. This is the client ID that is sent to the Kafka broker.
  • Use name as client ID
    If checked, then the name of the Kafka client will also be used as the client ID that is sent to the Kafka broker.
  • Enable metadata message processing
    When enabled, then messages consumed by the connector retain Kafka topic partition and offset information. This information can be passed along in Dataristix task pipelines. Once delivery of the message is complete, a separate Kafka topic may receive confirmation of the processed topic partition and offset, which in turn may be used to commit the topic partition offset in a consumer client (see example at the end of the chapter).
  • Bootstrap Servers
    Enter the one ore more Kafka broker hosts, including port if different from the default port of 9092.
  • Use encryption (TLS)
    Check this option to encrypt traffic between the client and the server (the broker must support encrypted connections).
  • Authentiction
    Select SASL or OAuth 2.0 authentication as required by your Kafka cluster. Enter credentials as required.

After configuring the above you can try to establish a connection to the cluster by pressing the “Test Connection” button.

Remove Kafka Client

Select the “Clients” navigation node and a Kafka client from the list on the right, then press the “Remove” toolbar button to remove the selected client. You can also right-click on a Kafka client and select “Remove” from the context menu.

../_images/kafka-remove-client.png

Warning

Note that all topics configured for the Kafka client will be also be removed. This affects all tasks using these topics.