Build Kafka topology with Tekton - Sat, Dec 5, 2020

Build Kafka topology with Tekton

Build Kafka topology with Tekton

Building a topology for a Kafka application can easily done suing the Topology builder as described here .
Combining the builder with Tekton Pipelines can create be the basis of a declarative GitOps style CI pipeline.
All code sample can be found in this Github repository.

The goals

The goal is to create a basic Tekton pipeline using Tekton and the Topology builder to provision a specific Kafka topology prior to deploying and testing a Kafka Streams application .

Pipeline overview

The pipeline contains of a relatevly simple task that provisions a Kafka cluster in a different namespace:

     +----------+                +
     |git source|    ns: ci      |   ns: kafka
     +-----+----+                |
           |                     |
           |                     |
           v                     |
+----------+---------+           |            +----------------+
|Task: build-topology|      +----+-----+      |Kafka:    +---+ |
|                    |      | SVC:     |      |Bootsrap  |   | |
|   +----------+     |  +-->+ kafka-   +----->+server    +---+ |
|   |Topology- |     |  |   | bootstrap|      |          +---+ |
|   |Builder   +--------+   +----+-----+      |          |   | |
|   +----------+     |           |            |          +---+ |
+--------------------+           |            +----------------+
                                 |
                                 +

The key points of the pipeline design are:

The details on how to create the above mentioned service are not part of this post but are described in detail here .

The build-topology task

The task has four parameters:

And the process of provisioning the cluster consists of two simple steps:

The following code example contains the important parts of the task:

1   params:
2     - name: kafka-bootstrap-host
3       type: string
4       default: kafka-bootstrap
5     - name: kafka-bootstrap-port
6       type: string
7       default: "9092"
8     - name: topology-file
9       type: string
10       default: topology.yaml
11     - name: topology-builder-properties-file
12       type: string
13       default: topology-builder.properties
14   steps:
15     - name: build-kafka-topology
16       image: purbon/kafka-topology-builder:1.0.0-rc.2
17       script: |
18         cd git-source
19         export BOOTSTRAP_SERVERS=$(params.kafka-bootstrap-host):$(params.kafka-bootstrap-port)
20         echo $BOOTSTRAP_SERVERS
21         sed -i 's/${BOOTSTRAP_SERVERS}/'$BOOTSTRAP_SERVERS'/g' $(params.topology-builder-properties-file)
22         cat $(params.topology-builder-properties-file)
23         kafka-topology-builder.sh \
24           --clientConfig $(params.topology-builder-properties-file) \
25           --topology $(params.topology-file) \
26           --allowDelete

The creation of the properties file is done using sed in place.

The pipeline and pipeline run

The pipeline is rather simple as it only encapsulates the task. In order to test the pipeline I used a basic topology with two topics. The complete pipeline run looks like this:\

1  ---
2  apiVersion: tekton.dev/v1beta1
3  kind: PipelineRun
4  metadata:
5    name: build-kafka-topology
6    namespace: ci
7  spec:
8    serviceAccountName: build-bot
9    pipelineRef:
10      name: build-kafka-topology
11    resources:
12    - name: git-source
13      resourceRef:
14        name: kafka-examples-git
15    params:
16    - name: topology-file
17      value: topology-builder/simple-example/topology.yaml
18    - name: topology-builder-properties-file
19      value: topology-builder/simple-example/topology-builder.properties

Note the supplied parameters in line 17 and 19.

Advantages

This approach combines two approaches: GitOps and cloud native CI with Tekton. The result increases the autonomy of the pipeline and fosters a declarative approach to software development.

Back to Home