Why Event-Driven Architecture Solves Real SaaS Scaling Problems
Event-driven systems decouple services so that the order service does not need to know about the billing service, the notification service, or the analytics service. Each service only publishes events to Kafka and subscribes to the events it cares about. This architectural style reduces deployment coordination and allows teams to scale, update, and fail independently.
In practice, a typical SaaS application sees order volume grow 10x during seasonal peaks. Synchronous REST calls between services create cascading failures and tight latency budgets. Kafka acts as a buffer, allowing the billing service to process events at its own pace while guaranteeing delivery even if the consumer is temporarily down.
KafkaJS 2.2 brings improved TypeScript support, better transaction handling, and native support for SASL and mTLS. Combined with Avro schemas stored in a Schema Registry, you gain compile-time safety and runtime compatibility checks that prevent producers from emitting breaking changes.
Local Development Environment with Docker Compose
Start with a reproducible local stack using Docker Compose. The following configuration includes Kafka, Zookeeper, Schema Registry, and Kafka UI. Save this as docker-compose.yml:
version: '3.8'
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.6.0
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- "2181:2181"
kafka:
image: confluentinc/cp-kafka:7.6.0
depends_on:
- zookeeper
ports:
- "9092:9092"
- "29092:29092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
schema-registry:
image: confluentinc/cp-schema-registry:7.6.0
depends_on:
- kafka
ports:
- "8081:8081"
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:9092
kafka-ui:
image: provectuslabs/kafka-ui:latest
ports:
- "8080:8080"
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry:8081
Run docker compose up -d. Verify the setup by visiting Kafka UI at http://localhost:8080. The Schema Registry will be available at http://localhost:8081. This stack matches the versions used in the production examples later in this article.
Choosing and Configuring a Schema Registry
Two practical options exist: Confluent Schema Registry (used in this guide) and Karapace. Confluent Schema Registry offers better UI integration and subject compatibility levels (NONE, BACKWARD, FORWARD, FULL). Karapace is lighter and runs without Zookeeper but has fewer enterprise features.
For most startups, Confluent Schema Registry running in the same Docker network is sufficient. Set the compatibility level to BACKWARD so consumers using older schema versions can still read events produced with newer versions that only add optional fields.
Register a schema using curl:
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data '{"schema": "{\"type\":\"record\",\"name\":\"OrderCreated\",\"fields\":[{\"name\":\"orderId\",\"type\":\"string\"},{\"name\":\"customerId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"createdAt\",\"type\":\"string\"}]}"}' \
http://localhost:8081/subjects/order-created-value/versions
This registers the order-created-value subject. The -value suffix follows the convention that separates key and value schemas.
Defining Avro Schemas and Generating TypeScript Types
Avro provides a compact binary format and schema evolution rules. Create schemas in a schemas/ directory. Here is a complete order-created.avsc:
{
"type": "record",
"name": "OrderCreated",
"namespace": "com.saas.events",
"fields": [
{ "name": "orderId", "type": "string" },
{ "name": "customerId", "type": "string" },
{ "name": "amount", "type": "double" },
{ "name": "currency", "type": "string", "default": "USD" },
{ "name": "items", "type": { "type": "array", "items": "string" } },
{ "name": "createdAt", "type": { "type": "long", "logicalType": "timestamp-millis" } }
]
}
Use avsc CLI or @kafkajs/confluent-schema-registry with TypeScript code generation. The package avro-to-ts can generate types automatically. Install it with:
npm install -D avro-to-ts npx avro-to-ts -i schemas/ -o src/types/events/
This produces a OrderCreated.ts file with proper types that you import throughout your services, eliminating stringly-typed event payloads.
Setting Up KafkaJS 2.2 Producer with Type Safety
Install the required packages:
npm install kafkajs @kafkajs/confluent-schema-registry avsc npm install -D @types/node typescript
Configure the producer in src/lib/kafka.ts:
import { Kafka, Partitioners } from 'kafkajs';
import { SchemaRegistry, SchemaType } from '@kafkajs/confluent-schema-registry';
const kafka = new Kafka({
clientId: 'order-service',
brokers: ['localhost:29092'],
retry: {
initialRetryTime: 300,
retries: 10,
maxRetryTime: 30000,
},
});
export const producer = kafka.producer({
createPartitioner: Partitioners.DefaultPartitioner,
idempotent: true, // enables exactly-once within a partition
transactionalId: 'order-tx', // required for transactions
transactionTimeout: 30000,
});
export const registry = new SchemaRegistry({
host: 'http://localhost:8081',
compatibility: 'BACKWARD',
});
export async function produceOrderCreated(event: OrderCreated) {
const payload = await registry.encode('order-created-value', event);
const result = await producer.send({
topic: 'order-created',
messages: [{
key: event.orderId,
value: payload,
headers: {
'event-type': 'OrderCreated',
'schema-version': '1',
},
}],
});
return result;
}
The idempotent: true setting ensures that even if the producer retries, duplicate messages are not written. The transactional producer (used later with the outbox pattern) builds on this foundation.
Implementing the Transactional Outbox Pattern
The outbox pattern solves the dual-write problem: you cannot atomically write to Postgres and publish to Kafka. Instead, write the event to an outbox table in the same database transaction as your business entity. A separate process (the outbox relay) reads pending events and publishes them to Kafka atomically using Kafka transactions.
Create the outbox table:
CREATE TABLE outbox (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
topic TEXT NOT NULL,
key TEXT,
value BYTEA NOT NULL,
headers JSONB DEFAULT '{}',
created_at TIMESTAMPTZ DEFAULT NOW(),
processed_at TIMESTAMPTZ,
error TEXT
);
Use Drizzle ORM (or Prisma) to insert into both the orders table and the outbox table inside one transaction. The relay service then marks events as processed only after successful Kafka transaction commit.
Here is the relay implementation using KafkaJS transactions:
async function relayOutbox() {
const tx = await producer.transaction();
try {
const pending = await db.query.outbox.findMany({
where: { processed_at: null, error: null },
limit: 50,
});
for (const event of pending) {
await tx.send({
topic: event.topic,
messages: [{
key: event.key,
value: event.value,
headers: event.headers,
}],
});
await db.update(outbox)
.set({ processed_at: new Date() })
.where({ id: event.id });
}
await tx.commit();
} catch (err) {
await tx.abort();
// mark failed events with error for retry or dead-lettering
}
}
Run this relay every 500ms using setInterval or better, use a proper scheduler such as BullMQ. This guarantees that business data and events stay consistent.
Building Robust Consumers with KafkaJS
Consumers must handle rebalancing, errors, and schema evolution gracefully. Create a consumer factory:
import { EachMessagePayload } from 'kafkajs';
export async function createConsumer(groupId: string, topics: string[]) {
const consumer = kafka.consumer({
groupId,
sessionTimeout: 25000,
heartbeatInterval: 3000,
maxBytesPerPartition: 1048576,
});
await consumer.connect();
await consumer.subscribe({ topics, fromBeginning: false });
await consumer.run({
eachMessage: async (payload: EachMessagePayload) => {
const decoded = await registry.decode(payload.message.value);
const eventType = payload.message.headers?.['event-type']?.toString();
try {
await handleEvent(eventType, decoded);
// manual commit only after successful business logic
} catch (error) {
// send to dead letter queue topic with original headers plus error metadata
await sendToDLQ(payload, error);
}
},
});
return consumer;
}
Use consumer groups per service. For the billing service, create a dedicated group billing-service-v1. This allows multiple instances to share the load via partition assignment. KafkaJS automatically handles rebalancing when instances join or leave.
Exactly-Once Semantics: What Actually Works
Kafka provides three delivery guarantees:
- At-most-once: Fast but can lose events.
- At-least-once: Default with retries but creates duplicates.
- Exactly-once: Requires idempotent producers, transactions, and the transactional outbox pattern.
In practice, combine idempotent producers (idempotent: true), transactional producers, and unique keys (orderId) for deduplication on the consumer side. Store the last processed event ID per aggregate in your database to achieve application-level exactly-once semantics.
Trade-off: Transactions add ~20-30ms latency per batch. For high-throughput services, batch events and use transactions only for critical money-related flows.
Observability: Metrics, Tracing, and Consumer Lag
Instrument KafkaJS with OpenTelemetry. Install @opentelemetry/instrumentation-kafkajs (community package) and configure it to emit spans for produce and consume operations.
Key metrics to collect:
- Producer throughput (messages/sec)
- Consumer lag per partition (critical SLA metric)
- Transaction commit latency
- Error rate per topic
Use Prometheus to scrape these metrics. A practical Grafana dashboard should show consumer lag exceeding 10,000 events as a critical alert. Implement a dead-letter queue topic (order-created-dlq) and monitor its size. Events in DLQ should trigger PagerDuty after 5 minutes.
Example consumer lag monitoring query using Kafka command line:
kafka-consumer-groups.sh --bootstrap-server localhost:29092 \ --describe --group billing-service-v1
Production teams run this periodically or integrate it into their observability platform.
Partitioning Strategy and Scaling Considerations
Choose partition keys carefully. For order events, use customerId as the key so all events for one customer land on the same partition. This preserves order per customer while allowing parallel processing across customers.
Start with 12 partitions for high-volume topics. KafkaJS 2.2 can handle consumers processing up to 15,000 events per second per instance on a standard 4-vCPU machine when using batch consumption.
When scaling consumers, add more instances before adding partitions. Re-partitioning a live topic is complex and usually requires a new topic and migration consumers. Plan partition count for 12-24 months of growth.
Complete Worked Example: Order to Billing Flow
Consider an e-commerce SaaS. When an order is placed:
- Order service saves order and inserts
OrderCreatedevent into outbox (single Postgres transaction). - Outbox relay publishes event transactionally to Kafka.
- Billing service consumes
order-created, creates an invoice, and publishesInvoiceCreatedevent. - Notification service consumes both events and sends emails.
Here is the billing consumer handler:
async function handleOrderCreated(event: OrderCreated) {
const existing = await db.query.invoices.findFirst({
where: { orderId: event.orderId }
});
if (existing) {
console.log(`Event ${event.orderId} already processed`);
return;
}
await db.transaction(async (tx) => {
const invoice = await createInvoice(tx, event);
const invoiceEvent: InvoiceCreated = {
invoiceId: invoice.id,
orderId: event.orderId,
customerId: event.customerId,
amount: event.amount,
createdAt: new Date(),
};
await tx.insert(outbox).values({
topic: 'invoice-created',
key: invoice.id,
value: await registry.encode('invoice-created-value', invoiceEvent),
headers: { 'event-type': 'InvoiceCreated' }
});
});
}
This pattern scales to dozens of services. Each service owns its own database and only communicates through well-defined events with schemas.
Production Deployment Options and Costs
For production, avoid self-managed Kafka unless you have a dedicated platform team. Three practical options exist in 2024:
- Confluent Cloud: Serverless mode starts at $0.55 per hour per CKU. Excellent Schema Registry and Stream Governance.
- Redpanda Cloud: Often 30-50% cheaper for the same throughput. Compatible with KafkaJS and Schema Registry API.
- AWS MSK: Good if you are already in AWS, but Schema Registry must be self-hosted or use Confluent.
Start with Confluent Cloud "Basic" cluster (single zone) for development and move to "Standard" with multiple availability zones for production. Enable client-side encryption with mTLS and use IAM-based authentication instead of API keys when possible.
Monitor cost drivers: partition count, retention period (default 7 days is often enough), and data transfer. A typical Series A SaaS with 2 million monthly events usually stays under $400/month on Confluent Cloud.
Schema Evolution Rules and Compatibility
Follow these practical rules when changing schemas:
- Add only optional fields with defaults.
- Never remove or rename fields in BACKWARD compatibility mode.
- Use FULL compatibility for high-stakes topics like payment events.
- Version your consumer groups when making breaking changes (
billing-service-v2). - Keep a schema changelog in your repository and review schema PRs the same way you review code.
The Schema Registry will reject producer registrations that violate the compatibility level, preventing runtime surprises.
Common Pitfalls and Production Hardening
Here are issues encountered in real implementations and their solutions:
- Duplicate processing: Always check database for existing records using the event
orderIdbefore processing. - Poison pills: Configure
max.poll.records=1during development and implement a circuit breaker that moves failing events to DLQ after 3 retries with exponential backoff. - Rebalancing storms: Increase
session.timeout.msto 45 seconds and use cooperative rebalancing (default in KafkaJS 2.2). - Large messages: Keep events under 1MB. For larger payloads, store the payload in S3 and emit only metadata in Kafka.
- Clock skew: Always use the producer timestamp from Kafka record headers instead of system time on consumers.
Implement a circuit breaker library that pauses consumption for a topic when error rate exceeds 5% for 2 minutes. This prevents one bad event from blocking the entire consumer group.
Testing Event-Driven Systems
Unit test individual handlers with mocked decoded events. For integration tests, use testcontainers to spin up real Kafka and Schema Registry instances:
import { KafkaContainer } from 'testcontainers';
const kafkaContainer = await new KafkaContainer()
.withExposedPorts(9093)
.start();
End-to-end tests should publish events to a test topic and assert correct side effects in the database and outbound events. Contract testing using Pact or Schemathesis against schemas stored in the registry catches breaking changes early.
Security Best Practices
Enable encryption at rest and in transit. Use SASL/SCRAM or mTLS for authentication. Implement ACLs so the order service can only write to order-created and read from payment-failed.
Never put PII in event keys (they are logged). For sensitive data, encrypt fields using envelope encryption before publishing and store keys in a separate secrets manager. Rotate keys periodically and include key identifier in event headers.
Audit all schema changes and consumer group membership. Confluent Schema Registry enterprise features provide subject-level access control that is worth the investment once you exceed 10 topics.
Migrating Legacy Synchronous Systems
Most companies do not start with event-driven architecture. The strangler fig pattern works well: keep the monolith but have it publish events for every state change. New services consume those events and gradually take over responsibilities.
Start with a single high-value flow (orders → billing) before expanding. Measure success by reduced coupling (number of direct HTTP calls between services) and improved deployment frequency of individual services.
Expect initial complexity to increase before it decreases. Invest in observability and schema governance from day one. Teams that skip schema registry usually regret it after their third breaking change.
This guide provides production-tested patterns used in systems processing millions of events daily. The combination of KafkaJS 2.2 transactional producers, the outbox pattern, Avro with Schema Registry, and careful observability creates a foundation that scales from prototype to public company infrastructure.
