Outbox Pattern with Kafka and NestJS: Ensuring Reliable Event-Driven Systems

When integrating microservices in remote systems, data consistency and dependability are crucial. Making sure that database transactions and event publishing occur atomically is one of the most difficult tasks. An established remedy for this issue is the Outbox Pattern.

In this post, we'll examine the Outbox Pattern using Kafka and NestJS and show you how to use it successfully in a practical setting.

What is the Outbox Pattern?

The Outbox Pattern is a design pattern that ensures reliable event publishing by storing events in a database table (Outbox) before processing them asynchronously. This prevents issues where a service commits a database transaction but fails to publish an event, leading to inconsistent state across systems.

How It Works

1️⃣ A service writes a domain event to an Outbox Table within the same database transaction as the business data.

2️⃣ A separate process (poller or message relay) reads the Outbox Table and publishes events to Kafka.

3️⃣ Once an event is successfully sent, it is marked as processed or deleted.

Use Case: Order Processing with Outbox Pattern in NestJS

Imagine an e-commerce application where an Order Service needs to reliably publish events to Kafka when a new order is created.

Step 1: Set Up the NestJS Application

First, create a NestJS project and install necessary dependencies:

nest new outbox-kafka-demo
cd outbox-kafka-demo
npm install @nestjs/microservices kafkajs @nestjs/typeorm typeorm pg

Step 2: Define the Order and Outbox Entities

Create an Order entity and an Outbox entity using TypeORM.

order.entity.ts

import { Entity, PrimaryGeneratedColumn, Column, CreateDateColumn } from 'typeorm';

@Entity()
export class Order {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  customerId: number;

  @Column()
  status: string;

  @CreateDateColumn()
  createdAt: Date;
}

outbox.entity.ts

import { Entity, PrimaryGeneratedColumn, Column, CreateDateColumn } from 'typeorm';

@Entity()
export class OutboxEvent {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  eventType: string;

  @Column({ type: 'json' })
  payload: any;

  @Column({ default: false })
  processed: boolean;

  @CreateDateColumn()
  createdAt: Date;
}

Step 3: Create an Order Service to Write to the Outbox Table

Instead of publishing to Kafka directly, orders are saved to the database along with an outbox event in the same transaction.

order.service.ts

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { Order } from './order.entity';
import { OutboxEvent } from './outbox.entity';

@Injectable()
export class OrderService {
  constructor(
    @InjectRepository(Order) private readonly orderRepo: Repository<Order>,
    @InjectRepository(OutboxEvent) private readonly outboxRepo: Repository<OutboxEvent>,
  ) {}

  async createOrder(customerId: number): Promise<Order> {
    const order = this.orderRepo.create({ customerId, status: 'PENDING' });

    const outboxEvent = this.outboxRepo.create({
      eventType: 'OrderCreated',
      payload: { orderId: order.id, customerId },
    });

    await this.orderRepo.save(order);
    await this.outboxRepo.save(outboxEvent);

    return order;
  }
}

Step 4: Implement a Poller to Process Outbox Events

A separate background job will read unprocessed events and publish them to Kafka.

outbox.processor.ts

import { Injectable, OnModuleInit } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { OutboxEvent } from './outbox.entity';
import { KafkaProducerService } from './kafka.producer.service';

@Injectable()
export class OutboxProcessor implements OnModuleInit {
  constructor(
    @InjectRepository(OutboxEvent) private readonly outboxRepo: Repository<OutboxEvent>,
    private readonly kafkaProducer: KafkaProducerService,
  ) {}

  async processEvents() {
    const events = await this.outboxRepo.find({ where: { processed: false } });

    for (const event of events) {
      await this.kafkaProducer.sendMessage('order-events', event.payload);
      event.processed = true;
      await this.outboxRepo.save(event);
    }
  }

  onModuleInit() {
    setInterval(() => this.processEvents(), 5000); // Run every 5 seconds
  }
}

Step 5: Kafka Producer to Publish Events

The Kafka Producer will send the message to the Kafka topic.

kafka.producer.service.ts

import { Injectable } from '@nestjs/common';
import { Kafka } from 'kafkajs';

@Injectable()
export class KafkaProducerService {
  private kafka = new Kafka({ brokers: ['localhost:9092'] });
  private producer = this.kafka.producer();

  async sendMessage(topic: string, message: any) {
    await this.producer.connect();
    await this.producer.send({
      topic,
      messages: [{ value: JSON.stringify(message) }],
    });
    await this.producer.disconnect();
  }
}

Step 6: Kafka Consumer to Process Events

A Kafka Consumer will listen to the order-events topic and process messages.

kafka.consumer.service.ts

import { Injectable, OnModuleInit } from '@nestjs/common';
import { Kafka } from 'kafkajs';

@Injectable()
export class KafkaConsumerService implements OnModuleInit {
  private kafka = new Kafka({ brokers: ['localhost:9092'] });
  private consumer = this.kafka.consumer({ groupId: 'order-group' });

  async onModuleInit() {
    await this.consumer.connect();
    await this.consumer.subscribe({ topic: 'order-events', fromBeginning: true });

    await this.consumer.run({
      eachMessage: async ({ message }) => {
        console.log('Received message:', message.value.toString());
      },
    });
  }
}

🤨 How This Ensures Reliability

✅ Atomicity: The order and outbox event are saved in one transaction.

✅ Retry Mechanism: If Kafka is down, the event stays in the database until it's processed.

✅ Guaranteed Delivery: Events will not be lost if the service crashes.

---

In NestJS applications that use Kafka, the Outbox Pattern is a potent way to guarantee dependable event-driven communication. By separating the database transaction from the message publishing process, we prevent inconsistencies and data loss.

To increase reliability when developing scalable microservices, think about using the Outbox Pattern.