Publishing Data to a Kafka Topic

We are now going to showcase how this telemetry data, can be integrated with Apache Kafka.

Step 1 - Stop the running docker containers

Stop all running containers by pressing Ctrl+c . We will restart these containers later with an updated gnmi.conf file.

Ctrl+C

  

Step 1 - Open the docker-compose.yml file if it was closed previously.

code -r /home/cisco/CiscoLive/DEVWKS-2135/docker-compose.yml

Step 2 - Update the new docker-compose.yml

Copy the content below to the docker-compose.yml file and press Ctrl+s to save the file.

version: "2.1"
services:
  telegraf:
    image: telegraf:1.22-alpine
    container_name: telegraf
    volumes:
      # Mount for telegraf config
      - ./telegraf/gnmi.conf:/etc/telegraf/telegraf.conf:ro
      - ./telegraf/certs/gnmi.crt:/etc/telegraf/certs/gnmi.crt:ro
    env_file:
      - influxv2.env
    ports:
      - 50050:50050

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - 9092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

volumes:
  influxdbv2:
  static-content:
  uwsgi:

    

      

Step 4 - Open gnmi.conf

Open file telegraf/gnmi.conf if it was closed previously

code -r /home/cisco/CiscoLive/DEVWKS-2135/telegraf/gnmi.conf

Step 5 - Replace the gNMI configuration file with this updated kafka output.

Replace the gnmi.conf file with the below content and press Ctrl+s to save the file.

[agent]
  logtarget = "file"
  logfile = "/var/log/telegraf/telegraf.log"
  debug = true
  hostname = "telegraf"
  interval = "10s"
  round_interval = false
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  
  
  [[inputs.gnmi]]
    # Address and port of the GNMI GRPC server
    addresses = ["10.15.24.10:50051", "10.15.24.11:50051","10.15.24.12:50051"]
    ## define credentials
    username = "admin"
    password = "cisco.123"
  
    ## enable client-side TLS and define CA to authenticate the device
    enable_tls = true
    tls_ca = "/etc/telegraf/certs/gnmi.crt"
    insecure_skip_verify = true
  
    # redial in case of failures after
    redial = "10s"
  [inputs.gnmi.tags]
    tag1 = "gnmi"
  
  [[inputs.gnmi.subscription]]
    name = "mac-address-table"
    origin = "openconfig"
    path = "/network-instances/network-instance/fdb/mac-table/entries"
    subscription_mode = "sample"
    sample_interval = "60s"
  
  
  [[outputs.file]]
    files = ["stdout"]
    data_format = "json"  
  
  [[outputs.kafka]]
      brokers = ["broker:29092"]
      ## Kafka topic for producer messages
      topic = "MACAddressTable"
      data_format = "json"
      namepass = ["mac-address-table"]
    

  

Step 6 - Restart the containers via Docker Compose

Run the docker-compose.yml file via the below command. Confirm that the subscriptions establish to the switches and the data is published to the Kafka broker.

docker-compose up --force-recreate

Step 7 - Validate that Telegraf can successfully publish to the Kafka topic

➜  DEVWKS-2135 ✗ | docker-compose up --force-recreate
Recreating telegraf  ... done
Recreating zookeeper ... done
Recreating broker    ... done
Attaching to telegraf, zookeeper, broker
broker       | ===> User
broker       | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
zookeeper    | ===> User
broker       | ===> Configuring ...
zookeeper    | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
zookeeper    | ===> Configuring ...
telegraf     | 2023-02-03T15:31:08Z I! Using config file: /etc/telegraf/telegraf.conf
.
.
.
telegraf     | 2023-02-03T15:31:08Z I! Starting Telegraf 1.22.3
telegraf     | 2023-02-03T15:28:25Z D! [sarama] producer/broker/1 starting up
telegraf     | 2023-02-03T15:28:25Z D! [sarama] producer/broker/1 state change to [open] on MACAddressTable/0
telegraf     | 2023-02-03T15:28:25Z D! [sarama] Connected to broker at broker:29092 (registered as #1)
telegraf     | 2023-02-03T15:28:25Z D! [outputs.kafka] Wrote batch of 26 metrics in 82.16105ms
telegraf     | 2023-02-03T15:28:25Z D! [outputs.kafka] Buffer fullness: 0 / 10000 metrics
telegraf     | 2023-02-03T15:28:35Z D! [outputs.kafka] Buffer fullness: 0 / 10000 metrics