How to Automate Kafka Topic Creation and Deletion, Using GitHub Actions and JavaScript

Automating Kafka topic creation and deletion is an important task for developers working with event-driven architectures. Whether you're managing a growing system or implementing infrastructure as code, manual topic management becomes unsustainable as your Kafka deployment grows. This tutorial shows you how to automate Kafka topic creation and deletion using JavaScript and GitHub Actions, making it part of your DevOps workflow.

The Challenge of Managing Kafka Topics

Topics are the foundation of event organization in Kafka. As your architecture grows, you'll create different topics to:

• Hold distinct types of events

• Store filtered versions of events

• Maintain transformed versions of the same events

Managing those topics manually becomes increasingly complex and error-prone as your system scales. Every new service might require new topics, and each environment (development, staging, production) needs its own set of consistently configured topics. Without automation, this manual overhead can slow down development and introduce configuration errors.

Why Automate Topic Management?

Automating Kafka topic creation and deletion offers several advantages:

1. Ensures consistent topic configuration across environments

2. Reduces human error in topic management

3. Enables version control of topic configurations

4. Streamlines DevOps workflows

5. Makes topic changes part of your CI/CD pipeline

Choosing the Right Tools for Automation

This tutorial uses JavaScript to create our automation solution. While you could use various languages, JavaScript offers several advantages for this task:

• Simplifies scripting compared to Bash

• Leverages existing JavaScript expertise in your team

• Provides excellent package ecosystem through npm

• Offers clean async/await syntax for handling Kafka operations

We’re going to implement this solution as a Node.js application running as a Kubernetes Job, making it perfect for teams using a VPC-accessible Kafka cluster.

Set Up The Application

The solution is a Node.js application and for that, you will need a Node.js project. You can create a new project using the npm init command. If you don't have Node.js and npm, you should download and install the required binaries from nodejs.org/en/download.

Open your terminal to the directory you want to create the app, then run the command npm init -y. Install the Kafka JavaScript client as a dependency using the command npm install kafkajs.

Implementing The Solution

Our automation approach consists of three main components:

1. A JSON configuration file that declares which topics to create or delete

2. A Node.js application that interfaces with Kafka to execute these operations

3. A GitHub Actions workflow that triggers the automation when changes are made to the JSON configuration file.

Let's build each component step by step.

Configuration File Setup

The application will read a list of topics to create or delete through a JSON file. This approach allows anyone to make changes to a JSON file in a GitHub repo and open a PR with their change. Once the PR is merged into the main branch, the code reads the data from that file and then creates or deletes a list of topics as desired.

Create a file named topics.json with the following content:

{
  "create": [],
  "delete": []
}

This structure provides a clear, version-controlled record of topic management operations. The create array will contain topics to be created, while the delete array specifies topics to be removed.

Building the Topic Management API

Create a file api.js with the following code:

async function createTopics(topics, kafkaAdmin) {
  if (topics.length > 0) {
    await kafkaAdmin.createTopics({
      topics: topics.map((topic) => ({
        topic,
        numPartitions: 1,        // Adjust based on your throughput needs
        replicationFactor: 3,    // Typically matches the number of brokers
        configEntries: [
          { name: "min.insync.replicas", value: "2" }  // Ensures write durability
        ],
      })),
    });
  }
}

async function deleteTopics(topics, kafkaAdmin) {
  if (topics.length > 0) {
    await kafkaAdmin.deleteTopics({ topics: topics });
  }
}

module.exports = { createTopics, deleteTopics };

This module exports functions to create and delete Kafka topics. The configuration values shown here are examples - you should adjust them based on your specific requirements for throughput, durability, and cluster size.

Creating the Main Application

Create a index.js file with the following code:

const { Kafka } = require("kafkajs");
const { createTopics, deleteTopics } = require("./api");
const topics = require("../topics.json");

const username = process.env.KAFKA_USERNAME;
const password = process.env.KAFKA_PASSWORD;
const brokers = process.env.KAFKA_URL ? process.env.KAFKA_URL.split(",") : [];

if (!username && !password && brokers.length === 0) {
  throw new Error("Missing Kafka Client Credential");
}

const kafka = new Kafka({
  clientId: "admin-script",
  brokers: brokers,
  ssl: {
    rejectUnauthorized: false,  // Commonly used in internal VPC setups
  },
  sasl: {
    mechanism: "scram-sha-512", // Note: KafkaJS also supports OAUTHBEARER for OAuth 2.0 authentication
    username,
    password,
  },
});

const admin = kafka.admin();

admin.connect().then(async () => {
  const existingTopics = await admin.listTopics();

  // Only create topics that don't already exist
  const newTopics = topics.create.filter((x) => !existingTopics.includes(x));
  await createTopics(newTopics, admin);

  // Only delete topics that currently exist
  const deletionTopics = topics.delete.filter((x) =>
    existingTopics.includes(x)
  );
  await deleteTopics(deletionTopics, admin);

  await admin.disconnect();
});

This main application handles the connection to Kafka, reads the topic configuration, and orchestrates the creation and deletion of topics. It includes safety checks to avoid duplicate operations and ensures clean connection management.

Automating Deployment with GitHub Actions

Now that we have our topic management application ready, let's integrate it into a continuous deployment pipeline. We'll use GitHub Actions to automatically run our topic management code whenever the topic configuration changes.

Add the file kafka.yml to the directory .github/workflows:

name: Deploy Kafka Topics Job

on:
  push:
    branches: [main]

env:
  JOB_NAME: kafka-topics
  AWS_REGION: eu-west-1
  KUBERNETES_CLUSTER: demo-cluster
  KUBERNETES_NAMESPACE: default

jobs:
  build-and-push:
    name: Build & Push to ECR
    runs-on: ubuntu-latest
    steps:
      - name: Git checkout
        uses: actions/checkout@v3

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-west-1

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - name: Add short commit hash
        id: short-commit-hash
        run: echo "sha_short=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Build Docker container and push to ECR
        uses: dfreilich/pack-action@v2.1.1
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ steps.short-commit-hash.outputs.sha_short }}
        with:
          args: "build ${{ env.ECR_REGISTRY }}/${{ env.JOB_NAME}}:${{ env.IMAGE_TAG}} --builder heroku/buildpacks --buildpack heroku/nodejs --publish"

  deploy-job:
    name: Deploy to Kubernetes
    needs: [build-and-push]
    runs-on: ubuntu-latest
    steps:
      - name: Git checkout
        uses: actions/checkout@v3

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: eu-west-1

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - name: Add short commit hash
        id: short-commit-hash
        run: echo "sha_short=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Set Image Name
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ steps.short-commit-hash.outputs.sha_short }}
        run: 'echo "IMAGE_NAME=$(echo ${ECR_REGISTRY})/$(echo ${JOB_NAME}):$(echo ${IMAGE_TAG})" >> $GITHUB_ENV'

      - name: Create Job
        env:
          SHA: ${{ steps.short-commit-hash.outputs.sha_short }}
        run: |
          aws eks update-kubeconfig \
            --region ${AWS_REGION} \
            --name ${KUBERNETES_CLUSTER}

          cat <<EOF | kubectl apply -f -
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: ${JOB_NAME}-${SHA}
            namespace: ${KUBERNETES_NAMESPACE}
            labels:
              jobgroup: ${JOB_NAME}
          spec:
            ttlSecondsAfterFinished: 259200
            template:
              spec:
                containers:
                - name: ${JOB_NAME}-${SHA}
                  image: ${IMAGE_NAME}
                  envFrom:
                  - secretRef:
                      name: kafka-secrets
                restartPolicy: Never
            backoffLimit: 2
          EOF

This workflow automates the entire deployment process. When changes are pushed to the main branch, it builds a container image of the Node.js application, pushes it to Amazon ECR, and creates a Kubernetes Job to execute our topic management code.

Conclusion

You've now learned how to automate Kafka topic creation and deletion using JavaScript. This automation approach using a JSON configuration file and GitHub Actions provides a scalable way to manage Kafka topics as your system grows. By integrating topic management into your CI/CD pipeline, you can ensure consistent and reliable topic configurations across your Kafka deployment.

The solution demonstrates how to transform manual topic management into a streamlined, automated process that fits well into modern DevOps practices. The code provides a foundation that you can build upon and customize for your specific needs.

The sample code is a mini-adaptation of what I used for a freelance project. Feel free to adapt it to your specific needs and automation requirements.

Need DevOps expertise or coaching? Contact me and I’d be pleased to support your team.
Find more content at devopsforjavascript.dev