Content Role: pillar

Event-Driven Architecture: Decoupling Microservices

Event sourcing and CQRS patterns with Kafka and EventBridge

Microservices promised independence, but most implementations create tightly coupled systems through synchronous HTTP calls. When Service A directly calls Service B, which calls Service C, you've built a distributed monolith with cascading failures, timeout chains, and deployment dependencies that negate the benefits of microservices architecture.

Event-driven architecture patterns solve this by inverting the dependency model. Services publish domain events without knowing who consumes them. Consumers subscribe to events they care about without coupling to producers. This article demonstrates how to implement event sourcing and CQRS patterns using modern tools like Apache Kafka and AWS EventBridge, with production-ready TypeScript examples.

Why Synchronous Communication Fails at Scale

Traditional request-response patterns create several critical problems in distributed systems:

Temporal coupling: Service A must wait for Service B to respond. If B is slow or down, A fails or times out. This creates a chain of dependencies where the slowest service determines overall system performance.

Availability coupling: Your system's availability becomes the product of all service availabilities. Three services with 99.9% uptime each yield 99.7% combined availability—unacceptable for critical systems.

Change amplification: Modifying Service B's API requires coordinating changes across all calling services. This coordination overhead grows exponentially with service count.

Scalability bottlenecks: Synchronous calls require both services to be available simultaneously. You can't independently scale producers and consumers based on their actual load patterns.

These issues compound in real-world scenarios. An e-commerce checkout might call payment, inventory, shipping, and notification services synchronously. If the notification service is slow, the entire checkout hangs—even though notifications aren't critical to order completion.

Event-Driven Architecture Fundamentals

Event-driven systems communicate through immutable facts about what happened. Instead of "create order," you publish "OrderCreated" events. Services react to these events independently, without direct coupling.

Three core patterns enable this architecture:

Event notification: Simple pub-sub where services emit events and others react. Useful for triggering side effects like sending emails or updating caches.

Event-carried state transfer: Events contain complete state snapshots, allowing consumers to maintain local copies without querying the producer. This eliminates read-time coupling.

Event sourcing: Store events as the source of truth rather than current state. Rebuild state by replaying events. This provides complete audit trails and temporal queries.

Implementing Event Sourcing with Kafka

Apache Kafka excels at event sourcing due to its log-based architecture, ordering guarantees, and retention capabilities. Here's a production-grade implementation:

import { Kafka, Producer, Consumer, EachMessagePayload } from 'kafkajs';

interface DomainEvent {
  eventId: string;
  eventType: string;
  aggregateId: string;
  aggregateType: string;
  timestamp: Date;
  version: number;
  payload: unknown;
  metadata: {
    userId: string;
    correlationId: string;
    causationId: string;
  };
}

class EventStore {
  private producer: Producer;
  private kafka: Kafka;

  constructor(brokers: string[]) {
    this.kafka = new Kafka({
      clientId: 'event-store',
      brokers,
      retry: {
        retries: 8,
        initialRetryTime: 100,
        maxRetryTime: 30000,
      },
    });
    this.producer = this.kafka.producer({
      idempotent: true,
      maxInFlightRequests: 5,
      transactionalId: 'event-store-producer',
    });
  }

  async append(event: DomainEvent): Promise<void> {
    const topic = `${event.aggregateType}-events`;

    await this.producer.send({
      topic,
      messages: [{
        key: event.aggregateId,
        value: JSON.stringify(event),
        headers: {
          'event-type': event.eventType,
          'correlation-id': event.metadata.correlationId,
        },
        timestamp: event.timestamp.getTime().toString(),
      }],
    });
  }

  async getEvents(
    aggregateId: string,
    aggregateType: string,
    fromVersion: number = 0
  ): Promise<DomainEvent[]> {
    const consumer = this.kafka.consumer({
      groupId: `replay-${aggregateId}-${Date.now()}`,
    });

    await consumer.connect();
    await consumer.subscribe({
      topic: `${aggregateType}-events`,
      fromBeginning: true,
    });

    const events: DomainEvent[] = [];

    await consumer.run({
      eachMessage: async ({ message }: EachMessagePayload) => {
        const event = JSON.parse(message.value!.toString()) as DomainEvent;
        if (event.aggregateId === aggregateId && event.version >= fromVersion) {
          events.push(event);
        }
      },
    });

    return events;
  }
}

This implementation uses Kafka's idempotent producer to prevent duplicate events and partitions by aggregate ID to maintain ordering guarantees. The transactional producer ensures atomic writes across multiple topics.

CQRS: Separating Reads from Writes

Command Query Responsibility Segregation (CQRS) pairs naturally with event sourcing. Write models optimize for consistency and business rules. Read models optimize for query performance and denormalization.

// Write side: Command handler
interface CreateOrderCommand {
  orderId: string;
  customerId: string;
  items: Array<{ productId: string; quantity: number; price: number }>;
}

class OrderAggregate {
  private events: DomainEvent[] = [];
  private state: {
    orderId: string;
    status: 'pending' | 'confirmed' | 'cancelled';
    totalAmount: number;
  } | null = null;

  static async load(
    orderId: string,
    eventStore: EventStore
  ): Promise<OrderAggregate> {
    const aggregate = new OrderAggregate();
    const events = await eventStore.getEvents(orderId, 'Order');

    events.forEach(event => aggregate.apply(event, false));
    return aggregate;
  }

  createOrder(command: CreateOrderCommand): void {
    if (this.state !== null) {
      throw new Error('Order already exists');
    }

    const totalAmount = command.items.reduce(
      (sum, item) => sum + item.price * item.quantity,
      0
    );

    this.apply({
      eventId: crypto.randomUUID(),
      eventType: 'OrderCreated',
      aggregateId: command.orderId,
      aggregateType: 'Order',
      timestamp: new Date(),
      version: 1,
      payload: {
        customerId: command.customerId,
        items: command.items,
        totalAmount,
      },
      metadata: {
        userId: command.customerId,
        correlationId: crypto.randomUUID(),
        causationId: command.orderId,
      },
    });
  }

  private apply(event: DomainEvent, isNew: boolean = true): void {
    switch (event.eventType) {
      case 'OrderCreated':
        this.state = {
          orderId: event.aggregateId,
          status: 'pending',
          totalAmount: (event.payload as any).totalAmount,
        };
        break;
      // Handle other event types
    }

    if (isNew) {
      this.events.push(event);
    }
  }

  getUncommittedEvents(): DomainEvent[] {
    return this.events;
  }
}

// Read side: Projection
class OrderReadModel {
  async handleOrderCreated(event: DomainEvent): Promise<void> {
    const payload = event.payload as any;

    // Store in optimized read database (PostgreSQL, MongoDB, etc.)
    await this.db.query(`
      INSERT INTO order_summary (
        order_id, customer_id, total_amount, status, created_at
      ) VALUES ($1, $2, $3, $4, $5)
    `, [
      event.aggregateId,
      payload.customerId,
      payload.totalAmount,
      'pending',
      event.timestamp,
    ]);
  }
}

The write model enforces business invariants and emits events. The read model subscribes to events and builds denormalized views optimized for queries. This separation allows independent scaling and technology choices for each side.

AWS EventBridge for Cross-Service Integration

While Kafka excels at high-throughput event streaming within your infrastructure, AWS EventBridge provides serverless event routing across AWS services and SaaS applications:

import { EventBridgeClient, PutEventsCommand } from '@aws-sdk/client-eventbridge';

class EventBridgePublisher {
  private client: EventBridgeClient;
  private eventBusName: string;

  constructor(region: string, eventBusName: string) {
    this.client = new EventBridgeClient({ region });
    this.eventBusName = eventBusName;
  }

  async publish(event: DomainEvent): Promise<void> {
    const command = new PutEventsCommand({
      Entries: [{
        EventBusName: this.eventBusName,
        Source: `order.${event.aggregateType}`,
        DetailType: event.eventType,
        Detail: JSON.stringify({
          ...event.payload,
          metadata: event.metadata,
        }),
        Time: event.timestamp,
      }],
    });

    const response = await this.client.send(command);

    if (response.FailedEntryCount && response.FailedEntryCount > 0) {
      throw new Error(`Failed to publish event: ${JSON.stringify(response.Entries)}`);
    }
  }
}

EventBridge rules route events to Lambda functions, Step Functions, SQS queues, or third-party destinations without code changes. This enables loose coupling between AWS services and external systems.

Common Pitfalls and Solutions

Event versioning: Events are immutable contracts. Use semantic versioning and upcasting to handle schema evolution:

interface EventUpcaster {
  canUpcast(event: DomainEvent): boolean;
  upcast(event: DomainEvent): DomainEvent;
}

class OrderCreatedV1ToV2Upcaster implements EventUpcaster {
  canUpcast(event: DomainEvent): boolean {
    return event.eventType === 'OrderCreated' && event.version === 1;
  }

  upcast(event: DomainEvent): DomainEvent {
    return {
      ...event,
      version: 2,
      payload: {
        ...(event.payload as any),
        currency: 'USD', // Add new required field with default
      },
    };
  }
}

Eventual consistency: Read models lag behind writes. Implement correlation IDs to track event processing and provide optimistic UI updates.

Event ordering: Kafka guarantees ordering within partitions. Always partition by aggregate ID to maintain consistency boundaries.

Duplicate events: Network failures cause retries. Make consumers idempotent by tracking processed event IDs or using natural idempotency keys.

Best Practices Checklist

• Use correlation and causation IDs to trace event chains across services
• Implement dead letter queues for failed event processing with alerting
• Version all events from day one using semantic versioning
• Partition strategically to balance load while maintaining ordering guarantees
• Monitor consumer lag to detect processing bottlenecks before they impact users
• Store events indefinitely or implement snapshot strategies for long-lived aggregates
• Test with chaos engineering to verify resilience under partition and failure scenarios
• Document event schemas in a shared registry with backward compatibility rules
• Implement circuit breakers for downstream dependencies in event handlers
• Use schema validation to catch incompatible events at publish time

Frequently Asked Questions

When should I use Kafka vs EventBridge?

Use Kafka for high-throughput internal event streaming (>10K events/sec), event sourcing, and when you need strong ordering guarantees. Use EventBridge for AWS service integration, lower-volume cross-service communication, and when you want serverless event routing without infrastructure management.

How do I handle transactions across multiple aggregates?

Don't. Event-driven systems embrace eventual consistency. Use sagas or process managers to coordinate multi-aggregate workflows through compensating events rather than distributed transactions.

What's the performance impact of event sourcing?

Reading requires replaying events, which can be slow for long-lived aggregates. Implement snapshots every N events to optimize load times. Write performance is excellent since you're appending to a log.

How do I query across multiple aggregates?

Build dedicated read models (projections) that denormalize data from multiple event streams. These read models can use any database optimized for your query patterns.

Should every service use event sourcing?

No. Event sourcing adds complexity. Use it for domains requiring audit trails, temporal queries, or complex business logic. Simple CRUD services can publish events without sourcing them.

How do I handle schema evolution in production?

Never remove or rename fields. Add new optional fields and use upcasters to transform old events to new schemas on read. Maintain multiple event versions simultaneously during transitions.

What about GDPR and data deletion?

Event sourcing conflicts with right-to-erasure requirements. Implement crypto-shredding (encrypt events with user-specific keys and delete keys) or use pseudonymization with separate mapping tables you can delete.

Event-driven architecture patterns transform microservices from distributed monoliths into truly independent, scalable systems. The initial complexity pays dividends in resilience, scalability, and maintainability as your system grows.