Kafka Devices

The Kafka Devices configuration page defines connections to Apache Kafka brokers. It is opened with Kafka (AWS/Azure) in the Configuration menu. Each Kafka device represents one broker connection with its own authentication settings, topics, key names, and message format.

A Kafka device exchanges variable values with its broker in two directions:

• Producing: every variable linked to the Kafka device is sent to the Producer Topic on each value change, in the message format set by Produce Format.
• Consuming: when Consumer is switched on, the Kafka device reads the Consumer Topic and writes the received values to linked variables that are Writeable.

A variable is linked to a Kafka device on its own configuration page, with the Kafka Device field on the Gateway tab of the variable dialog. Variables of every device protocol and Local Variables can be linked. The Kafka Variables page lists the variables linked to a Kafka device.

Device Settings

Create a new device with the Add button in the toolbar. The Add device dialog contains all device settings organized in the sections General, Connection, Topics, Consumer Settings, Producer Settings, and Diagnostics, with default values already filled in. Enter at least the Name and Bootstrap Servers and press Create.

The device row shows Enable, Name, Bootstrap Servers, Sasl Mechanism, Security Protocol, SASL Username, SASL Password, Consumer, and Description directly, and these values can be edited in place. The remaining settings can be shown as columns with the Columns button. Press the Settings icon on a device row to open the Edit device dialog, which contains the same sections and all settings of the device. Press Save to apply the changes. The Variables icon opens the Kafka Variables page of the device, and the Events icon opens the System Log entries of the device.

Security Protocol and Sasl Mechanism are applied as configured, to the producer and to the consumer. With Plaintext or Ssl, the connection uses no SASL authentication and SASL Username and SASL Password may stay empty. With SaslPlaintext or SaslSsl, the Sasl Mechanism, SASL Username, and SASL Password are used. In the device row and through the Configuration API, the spellings SASL_PLAINTEXT, SASL_SSL, and SCRAM-SHA-512 are accepted as well. A value that cannot be used is refused when the device is saved.

Linking Variables

A variable is linked to a Kafka device on the Gateway tab of its variable dialog. The Gateway tab holds Modbus Address, OPC UA, Writeable, MQTT Publisher, Kafka Device, Kafka Remap Device, and Kafka Remap Variable. The variable grids offer the same fields as columns through the Columns button.

In Kafka messages, every linked variable is identified by a device name and a variable name. Without remaps, the device name is the device that owns the variable, or Local for a Local variable, and the variable name is the name of the variable without the device name and dot in front of it. A variable PLC1.Pump1_Speed on device PLC1 and a variable Pump1_Speed on device PLC1 are both identified as device PLC1 and variable Pump1_Speed. A Local variable Counter is identified as device Local and variable Counter. Kafka Remap Device and Kafka Remap Variable replace these names when they are set.

A linked variable is produced, and can be written by consumed messages, while the variable and its device are enabled. On a variable created by a UDT instance, Kafka Device, Kafka Remap Device, and Kafka Remap Variable are owned by the UDT definition and are set on the variable member.

Renaming a Kafka device, in the device row or in the Edit device dialog, updates the Kafka Device field of every variable and UDT variable member that names the device in the same operation, together with the UDT parameter values and parameter defaults that hold the old name and are used only to feed the Kafka Device field of members.

Produced Messages

The producer runs when Producer Topic and Produce Format are both set. Each value change of a linked variable produces one message to the Producer Topic, in the order of the changes and without a message key.

Produce Format is a text template. The tokens are replaced as plain text, without JSON escaping, so the template determines the quotes around each value.

TimeStamp Format determines the {TimeStamp} token:

• TimeStamp Format empty: the time stamp of the value as Unix epoch time in milliseconds (UTC), for example 1791547200000.
• TimeStamp Format set: the current UTC time at the moment the message is produced, formatted with the .NET date and time format. The format yyyy-MM-dd HH:mm:ss gives 2026-10-09 12:00:00. A format that cannot be applied sends the default date and time text and adds an entry to the log of the device.

{"device":"{DeviceName}","variable":"{VariableName}","timestamp":"{TimeStamp}","value":"{VariableValue}"}

{"device":"PLC1","variable":"Pump1_Speed","timestamp":"1791547200000","value":"1450"}

While the broker cannot be reached, messages wait in the queue of the Kafka device, up to Max Forwarding Queue Size on the Parameters page. When the queue is full, the oldest messages are dropped and the System Log reports it. Messages still in the queue when the device stops are handed to the broker before the producer closes.

Consumed Messages

With Consumer on, the Kafka device reads the Consumer Topic as a member of the consumer group set by Group Id. Consumer Topic, Group Id, Client Id, Device Key, Variable Key, and Value Key must all be set for the consumer to start. When one of them is empty, the log of the device names it and System Metrics shows the device in the Error state.

A consumed message is a JSON object. Its top-level properties named by Device Key, Variable Key, and Value Key hold the device name, the variable name, and the value. A message that is not valid JSON, or that lacks one of these properties, is rejected and logged.

{"device":"PLC1","variable":"Pump1_Setpoint","value":"1200"}

The device name and the variable name of the message are matched against the variables linked to this Kafka device, by the rules below in this order. The first rule that matches determines the variable that is written.

When one rule matches several linked variables, the variable that comes first in the sorted list of full variable names is written, and the log of the device reports the ambiguity at Verbose 1. A message that matches no enabled variable is logged as unavailable.

The matched variable is written when all of the following apply:

• Consumer is on for the Kafka device.
• The variable is Writeable.
• A Local variable has External Source enabled.
• The variable and its device are enabled.
• The server is the primary server.

The values true and false, in any letter case, are written as 1 and 0. Every accepted write is recorded as a historic event with the Identity Kafka and a Message such as SetValue PLC1.Pump1_Setpoint 1200 (KafkaBroker1, plant.commands): the variable, the value, the Kafka device, and the topic. The event is shown in the Event Viewer. A write refused because the variable is not Writeable is logged the first time for each variable, and every refused write is logged at Verbose 1.

A consumer group without committed offsets, for example a new Group Id, starts reading at the latest offset of the Consumer Topic. Messages published before the group first connected are therefore not written, so old commands are never replayed as writes. A group with committed offsets continues from its last committed offset.

Kafka Variables

Press the Variables icon on a device row to open the Kafka Variables page of that device. The page is read-only and lists every enabled variable whose Kafka Device field names the device, from all device protocols and from Local Variables. Links and remaps are changed on the configuration page of the variable.

Use Search, the column filters, and the Columns button to narrow the list. The Live Value column is updated while the page is open.

TLS Certificates

With Security Protocol Ssl or SaslSsl, the broker certificate is verified against the trusted CAs of the operating system. A broker with a certificate from a public CA needs no further settings.

For a broker with a certificate from a private CA, enter the subject name of that CA certificate in CA Certificate Name. The broker certificate is then verified against that CA only. For a test broker with a self-signed certificate, switch on Allow Untrusted Certificates instead. For a broker that requires client certificates, enter the subject name of the client certificate in Client Certificate Name.

The certificates are looked up by subject name on the server that runs EnvisionSCADA. They are not part of the configuration, so install them on each server, including the standby server.

The device events show which certificates were found and used each time the device connects, and the reason when a certificate cannot be used. The device then connects without that certificate and the broker refuses the connection.

Notes

Kafka devices run on the primary server only, and the standby server keeps them in the Standby state. After a switchover, the new primary server starts the producer and the consumer of every enabled Kafka device.

The state, the queue, and the number of linked variables of every Kafka device are shown in the Messaging and forwarding panel of the System Metrics page.

Enable Verbose only during commissioning or diagnostics. It increases log volume significantly.

Upgrading from Earlier Versions

Earlier versions configured Kafka per variable with the Kafka Producer and Kafka Consumer settings on a Kafka tab of the variable dialog. These two settings are replaced: a linked variable is always produced, and consumed writes are enabled per Kafka device with Consumer and per variable with Writeable. Kafka Device, Kafka Remap Device, and Kafka Remap Variable keep their meaning and are on the Gateway tab.

The project is converted automatically at the first start after the upgrade. The same conversion runs when a configuration database of an earlier version is loaded, for example from a project transfer or a backup, and when a Git repository or a UDT type file of an earlier version is imported. Each variable with a Kafka Device is converted as follows:

• An IEC104 variable with a monitor direction object type cannot be Writeable. When it had Kafka Consumer on, Writeable stays off and the case is logged, so consumed messages for this variable are refused.
• Every Kafka device with at least one variable that had Kafka Consumer on gets Consumer switched on.
• Earlier versions always connected with ScramSha512 and SaslSsl, whatever Sasl Mechanism and Security Protocol held. The upgrade sets both settings to these values on every Kafka device and logs the previous values, so the connection stays the same. From this version on, the configured values are applied.
• UDT variable members are converted with the same rules. A linked member with Kafka Consumer true, or set by a parameter, gets Writeable ticked, and a linked member with Kafka Producer and Kafka Consumer both false or empty gets its Kafka Device cleared. A member with an IEC104 object type that is not a fixed command type keeps Writeable off. An override of Kafka Consumer becomes an override of Writeable and Kafka Device, and an override of Kafka Producer becomes an override of Kafka Device. When the configuration database is converted, Writeable or Kafka Device is overridden on a materialized variable where the next push would otherwise undo the conversion of that variable.
• A consumer group without committed offsets starts at the latest offset of the Consumer Topic. Earlier versions started such a group at the earliest offset.
• With Produce Format empty, nothing is sent. Earlier versions sent an empty message for every value change.

Every change is listed in the migration log. Each line starts with the text Kafka migration and is written to the System Log with the origin DatabaseHelper and to the file ConfigV73-Kafka-[date]-[time].log in the Migrations subfolder of the File Logging Path directory. With File Logging Path empty, this is the Logs\Migrations folder in the EnvisionSCADA application directory. The file is written only when the conversion changed something, and log retention leaves it in place.

A Git import of an earlier version lists the same lines in the import details and writes them to GitImport-Kafka-[date]-[time].log in the same folder. A UDT type import of an earlier version lists them with the warnings of the import.