Build Kafka topology declaratively

Build Kafka topology declaratively

Developing streams applications and testing them usually requires having a specific topology of topics, partitions etc. in place.

The imperative approach

One way of creating the such a set of topics can be done using the appropriate scripts that come with a Kafka installation. For example when using strimizi on kubernetes a topic can be created using the following example:

kubectl -n kafka run kafka-producer -ti \
    --image=strimzi/kafka:0.20.0-kafka-2.6.0 \
    --rm=true --restart=Never \
    -- bin/kafka-topics.sh --create --topic product-updates --bootstrap-server my-cluster-kafka-bootstrap:9092

Usually a series of commands and scripts like the one above need to be run in a specific sequence to create the whole topology.\

Introducing the declarative approach

While this imperative approach works, there is also a declarative approach using the Kafka Topology Builder .
The declarative approach differs in that the desired topology is described in a descriptor file rather then using a series of commands.

A very simple example looks like this:

---
context: "kafka-examples"
projects:
  - name: "order-example"
    topics:
      - name: "incoming-orders"
        config:
          replication.factor: "1"
          num.partitions: "3"
      - name: "finished-orders"
        config:
          replication.factor: "1"
          num.partitions: "3

This topology can be applied to a Kafka cluster using the docker image of the Topology builder like this:

docker run -v ${PWD}:/example \
    purbon/kafka-topology-builder:1.0.0-rc.2 \
    kafka-topology-builder.sh \
    --clientConfig /example/topology-builder.properties \
    --topology /example/topology.yaml \
    --allowDelete

Not that depending on where the topology file is located and where the image is run, the command might look different.

Advantages

Describing a topology like this aids software delivery techniques like GitOps where there is an ongoing convergence between the deployment and source control. Thus changed can easily be rolled back and every change is tracked under source control.
In addition the model described in the file introduces an abstraction from the actual means by which the model is applied.

Application

As described here Jenkins and Travis could be used for the convergence. But it also seems that cloud native CI/CD tools like argo and Tekton could be good fit in a cloud environment.

Back to Home