Transactional Outbox Pattern: Reliable Event Publishing in Distributed Systems

Every event-driven service eventually has to do two things at once: change its own state and tell the rest of the system about it. A database write and a broker publish cannot be made atomic, so one can fail while the other succeeds. What is left behind is an order nobody reserved stock for, or reserved stock for an order that does not exist.

The Transactional Outbox Pattern removes the second write. The event goes into an outbox table in the same database, inside the same transaction as the business data, and a separate relay reads that table and publishes. Start with a polling relay: it needs no new infrastructure and its behavior is easy to follow in SQL. Move to Change Data Capture or a managed stream-to-bus pipe when poll latency or database load becomes the constraint. All three relays deliver at-least-once, so consumer-side idempotency is required either way.

The Dual-Write Problem

Consider a common scenario: an order service needs to save an order to the database and publish an OrderCreated event. The naive approach looks like this:

async function createOrder(orderData: Order) {
  // Step 1: Save to database
  await db('orders').insert(orderData);

  // Step 2: Publish event
  await messageQueue.publish('OrderCreated', orderData);
}

Two failure modes account for most of the damage, and both leave the system inconsistent.

Failure Scenario 1: Database succeeds, event publish fails

• Network timeout to message broker
• Message broker temporarily down
• Your service crashes after database write
• Result: Order exists in database, but inventory service never receives the event. Stock is never reserved.

Failure Scenario 2: Event publish succeeds, database fails

• Database write violates constraint
• Transaction rolled back due to deadlock
• Database connection lost
• Result: Inventory service receives event and reserves stock, but order doesn’t exist. Data inconsistency.

Why Not Use Two-Phase Commit (2PC)?

Distributed transactions can technically span a database and a broker. The trade-offs make 2PC impractical for this problem:

• Performance overhead: Coordinating transactions across systems adds significant latency
• Reduced availability: If any participant is down, the entire operation fails
• Complexity: Implementing XA transactions correctly is difficult
• Limited support: Many message brokers don’t support 2PC
• Coupling: Violates microservices independence principles

Avoiding distributed transactions is better than trying to make them work reliably.

Understanding the Outbox Pattern

Instead of writing to two separate systems (database + message broker), the pattern writes to two tables in the same database within a single ACID transaction.

Core Components

1. Outbox Table: Stores events to be published, lives in the same database as your business data
2. Business Transaction: Single ACID transaction writing to both business tables and outbox
3. Message Relay: Separate process reads outbox and publishes to message broker
4. Idempotent Consumers: Downstream services handle duplicate events correctly

How It Works

Once the commit succeeds, the relay is free to crash, retry, or fall behind without threatening correctness. The worst it can do is deliver late or deliver twice, and both are recoverable.

Implementation Approach 1: Polling Publisher

The simplest approach polls the outbox table periodically. Here’s what works in practice:

Basic Implementation

-- Outbox table schema (PostgreSQL)
CREATE TABLE outbox (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_type VARCHAR(100) NOT NULL,
  aggregate_id VARCHAR(100) NOT NULL,
  event_type VARCHAR(100) NOT NULL,
  payload JSONB NOT NULL,
  created_at TIMESTAMP DEFAULT NOW(),
  published BOOLEAN DEFAULT FALSE
);

-- Critical index for efficient polling
CREATE INDEX idx_outbox_unpublished
ON outbox(created_at)
WHERE published = false;

Producer: Write to Outbox

async function createOrder(orderData: Order) {
  await db.transaction(async (trx) => {
    // Insert order
    const order = await trx('orders').insert({
      id: orderData.id,
      customer_id: orderData.customerId,
      total: orderData.total,
      status: 'PENDING'
    }).returning('*');

    // Insert event to outbox IN SAME TRANSACTION
    await trx('outbox').insert({
      id: uuid(),
      aggregate_type: 'Order',
      aggregate_id: order[0].id,
      event_type: 'OrderCreated',
      payload: {
        orderId: order[0].id,
        customerId: orderData.customerId,
        total: orderData.total,
        items: orderData.items
      },
      created_at: new Date()
    });

    // Both succeed or both fail - atomicity guaranteed
  });
}

Publisher: Poll and Publish

async function publishOutboxEvents() {
  // Use FOR UPDATE SKIP LOCKED to prevent concurrent processing
  const events = await db.raw(`
    SELECT * FROM outbox
    WHERE published = false
    ORDER BY created_at
    LIMIT 100
    FOR UPDATE SKIP LOCKED
  `);

  for (const event of events.rows) {
    try {
      // Publish to message broker
      await messageQueue.publish(event.event_type, {
        messageId: event.id,  // Important for deduplication
        aggregateId: event.aggregate_id,
        payload: event.payload
      });

      // Mark as published
      await db('outbox')
        .where('id', event.id)
        .update({ published: true });

    } catch (error) {
      console.error('Failed to publish event:', error);
      // Will retry on next poll - at-least-once delivery
    }
  }
}

// Run publisher every 5 seconds
setInterval(publishOutboxEvents, 5000);

The FOR UPDATE SKIP LOCKED clause is critical: it prevents multiple publisher instances from processing the same events, enabling horizontal scaling.

Polling Pros and Cons

Pros:

• Simple to implement and understand
• No additional infrastructure required
• Works with any database
• Easy to debug with SQL queries

Cons:

• Polling adds database load
• Latency depends on poll interval (5-10 seconds typical)
• Less efficient than CDC for high volumes

Implementation Approach 2: Change Data Capture (CDC)

For production systems at scale, CDC eliminates polling overhead by monitoring the database transaction log directly.

How CDC Works

Instead of polling the outbox table, CDC tools like Debezium monitor the database’s Write-Ahead Log (PostgreSQL) or Binary Log (MySQL). When an outbox event is written, the CDC tool detects it and publishes to your message broker automatically.

PostgreSQL + Debezium Setup

-- 1. Enable logical replication (requires PostgreSQL restart)
ALTER SYSTEM SET wal_level = 'logical';
-- Note: PostgreSQL must be restarted for wal_level change to take effect

-- 2. Create publication for outbox table
CREATE PUBLICATION outbox_publication FOR TABLE outbox;

-- 3. Grant replication rights to Debezium user
ALTER USER debezium_user WITH REPLICATION;

Debezium Configuration

{
  "name": "outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres.example.com",
    "database.port": "5432",
    "database.user": "debezium_user",
    "database.password": "${DB_PASSWORD}",
    "database.dbname": "orders_db",
    "database.server.name": "orders",
    "table.include.list": "public.outbox",
    "plugin.name": "pgoutput",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.table.field.event.type": "event_type",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.payload": "payload"
  }
}

Producer Code (Identical to Polling)

The beauty of CDC: your application code doesn’t change. You still write to the outbox table in the same transaction. Debezium handles the publishing.

// Same code as polling approach - no changes needed
async function createOrder(orderData: Order) {
  await db.transaction(async (trx) => {
    await trx('orders').insert(orderData);
    await trx('outbox').insert({
      aggregate_type: 'Order',
      aggregate_id: orderData.id,
      event_type: 'OrderCreated',
      payload: orderData
    });
  });
  // Debezium automatically detects the new outbox row and publishes
}

CDC Pros and Cons

Pros:

• Near real-time event publishing (< 1 second)
• Minimal database overhead (reads WAL, not tables)
• Throughput scales with the log reader and the broker instead of with table scans
• Preserves event order per partition

Cons:

• Complex infrastructure (Kafka Connect, Debezium)
• Requires operational expertise
• Database-specific setup (WAL configuration)
• More expensive than serverless options

AWS Implementation: DynamoDB + EventBridge Pipes

AWS provides a serverless outbox implementation using DynamoDB Streams and EventBridge Pipes. This is the recommended approach for AWS-native architectures.

Architecture

Implementation

// 1. Write both items in single transaction
// Note: DynamoDB transactions have limits - max 100 items, 4MB aggregate size
async function createOrder(orderData: Order) {
  await ddb.send(new TransactWriteItemsCommand({
    TransactItems: [
      {
        Put: {
          TableName: 'Orders',
          Item: {
            orderId: { S: orderData.id },
            customerId: { S: orderData.customerId },
            total: { N: orderData.total.toString() },
            status: { S: 'PENDING' }
          }
        }
      },
      {
        Put: {
          TableName: 'Outbox',
          Item: {
            eventId: { S: uuid() },
            aggregateType: { S: 'Order' },
            aggregateId: { S: orderData.id },
            eventType: { S: 'OrderCreated' },
            payload: { S: JSON.stringify(orderData) },
            timestamp: { N: Date.now().toString() }
          }
        }
      }
    ]
  }));
}

Infrastructure as Code (AWS CDK)

import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import * as pipes from 'aws-cdk-lib/aws-pipes';
import * as events from 'aws-cdk-lib/aws-events';

// 1. Create outbox table with streams enabled
const outboxTable = new dynamodb.Table(this, 'OutboxTable', {
  partitionKey: { name: 'eventId', type: dynamodb.AttributeType.STRING },
  stream: dynamodb.StreamViewType.NEW_IMAGE,  // Critical: stream new items
  billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
  removalPolicy: cdk.RemovalPolicy.DESTROY
});

// 2. Create event bus
const eventBus = new events.EventBus(this, 'OrderEventBus', {
  eventBusName: 'order-events'
});

// 3. Create EventBridge Pipe (NO Lambda needed!)
new pipes.CfnPipe(this, 'OutboxPipe', {
  source: outboxTable.tableStreamArn!,
  target: eventBus.eventBusArn,
  roleArn: pipeRole.roleArn,
  sourceParameters: {
    dynamoDbStreamParameters: {
      startingPosition: 'LATEST',
      batchSize: 10,
      maximumRetryAttempts: 3,  // Note: Default is -1 (infinite retry)
      deadLetterConfig: {
        arn: dlqQueue.queueArn
      }
    }
  },
  targetParameters: {
    eventBridgeEventBusParameters: {
      detailType: 'OutboxEvent',
      source: 'outbox.publisher'
    }
  }
});

Why This Approach Works

No Lambda code for publishing: EventBridge Pipes automatically reads DynamoDB Streams and publishes to EventBridge. This eliminates:

• Cold start latency
• Lambda billing for publisher
• Code to maintain for the relay

Built-in reliability: Pipes include retry logic, dead-letter queues, and monitoring out of the box.

Cost efficiency: Billing follows event volume, and there is no idle publisher to pay for.

Cost Analysis

At us-east-1 list prices, a workload of 10 million events per month costs:

• DynamoDB Streams: billed per GetRecords call at $0.02 per 100,000 stream read request units. At a batch size of 10 that is roughly 1 million calls, under $1/month. The Lambda-trigger exemption does not cover Pipes.
• EventBridge Pipes: $0.40 per million requests = $4.00/month
• EventBridge custom event bus: $1.00 per million events = $10.00/month
• Total: roughly $14/month, on top of the DynamoDB writes you were already paying for

A polling relay has no per-event fee, so it always looks cheaper on the invoice. The cost moves rather than disappears: the poller runs whether or not events exist, every poll is a query against the production database, and the relay is code someone has to own, monitor, and get paged for. Choose between the two on operational load, not on the per-million line.

Handling Ordering and Idempotency

Ordering Guarantees

The outbox pattern preserves ordering per partition, not globally across all events.

// Ensure events for same aggregate are ordered
await kafka.producer.send({
  topic: 'order-events',
  messages: [{
    key: event.aggregateId,  // All events for ORDER-123 go to same partition
    value: JSON.stringify(event.payload)
  }]
});

For DynamoDB Streams, use the aggregate ID as the partition key:

await dynamodb.put({
  TableName: 'Outbox',
  Item: {
    aggregateId: 'ORDER-123',  // Partition key - ensures ordering
    eventId: uuid(),  // Sort key
    eventType: 'OrderCreated',
    timestamp: Date.now()
  }
});

The Inbox Pattern: Consumer-Side Idempotency

The outbox pattern guarantees at-least-once delivery, which means events may be delivered multiple times. Consumers must handle duplicates.

The Inbox Pattern provides idempotent processing:

async function handleOrderCreatedEvent(event: OrderCreatedEvent) {
  await db.transaction(async (trx) => {
    // 1. Check if already processed
    const existing = await trx('inbox')
      .where('message_id', event.messageId)
      .first();

    if (existing) {
      console.log('Duplicate message, skipping:', event.messageId);
      return;  // Idempotent - safe to skip
    }

    // 2. Process the event (your business logic)
    await trx('inventory')
      .where('product_id', event.productId)
      .decrement('quantity', event.quantity);

    // 3. Record as processed IN SAME TRANSACTION
    await trx('inbox').insert({
      message_id: event.messageId,
      event_type: event.type,
      processed_at: new Date()
    });

    // Either all three operations succeed, or all fail
  });

  // ACK to message broker only after successful commit
  await messageQueue.ack(event.messageId);
}

Inbox table schema:

CREATE TABLE inbox (
  message_id UUID PRIMARY KEY,
  event_type VARCHAR(100),
  processed_at TIMESTAMP DEFAULT NOW(),
  payload JSONB  -- Optional: for debugging
);

-- Cleanup old processed messages (run daily)
DELETE FROM inbox
WHERE processed_at < NOW() - INTERVAL '7 days';

Complete Pattern: Outbox + Inbox

Performance Considerations

Database Performance

Outbox table growth: Without cleanup, the outbox table grows indefinitely, causing significant performance degradation.

-- Strategy 1: Delete immediately after publish
DELETE FROM outbox WHERE id = $1 AND published = true;

-- Strategy 2: Batch cleanup (run daily via cron)
DELETE FROM outbox
WHERE published = true
  AND created_at < NOW() - INTERVAL '7 days';

-- Strategy 3: Table partitioning (PostgreSQL 10+)
CREATE TABLE outbox_2025_12 PARTITION OF outbox
  FOR VALUES FROM ('2025-12-01') TO ('2026-01-01');

-- Drop old partitions (much faster than DELETE)
DROP TABLE outbox_2025_11;

Index optimization: The partial index only indexes unpublished events, saving space:

CREATE INDEX idx_outbox_unpublished
ON outbox(created_at)
WHERE published = false;

Polling Publisher Tuning

Poll interval trade-offs:

• 1 second: Low latency, high database load
• 5 seconds: Balanced (recommended for most cases)
• 10+ seconds: Low overhead, higher latency

Batch size:

// Too small: many queries, inefficient
const batchSize = 10;

// Too large: long transactions, lock contention
const batchSize = 10000;

// Optimal: balance efficiency and transaction length
const batchSize = 100;  // Recommended starting point

CDC Performance

Monitor replication lag to ensure Debezium keeps up:

-- Check replication slot lag
SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as lag
FROM pg_replication_slots
WHERE slot_type = 'logical';

If lag grows, WAL files accumulate and can fill disk. This is a critical operational concern in production CDC setups.

Common Pitfalls and Solutions

Pitfall 1: Unbounded Table Growth

Problem: Outbox table grows indefinitely, queries slow down.

Solution: Implement automatic cleanup in your publisher:

async function publishAndCleanup() {
  // Publish events
  await publishOutboxEvents();

  // Cleanup old published events (every 100 iterations)
  if (cleanupCounter++ % 100 === 0) {
    await db('outbox')
      .where('published', true)
      .where('created_at', '<', db.raw("NOW() - INTERVAL '7 days'"))
      .delete();
  }
}

Pitfall 2: Message Relay Failure Goes Unnoticed

Problem: Publisher crashes, events pile up unpublished.

Solution: Monitor outbox age metrics:

async function checkOutboxHealth() {
  const result = await db('outbox')
    .where('published', false)
    .min('created_at as oldest')
    .first();

  if (!result.oldest) return;  // No unpublished events

  const ageMs = Date.now() - new Date(result.oldest).getTime();
  const ageMinutes = ageMs / 60000;

  if (ageMinutes > 5) {
    alerting.trigger('OUTBOX_LAG_HIGH', {
      ageMinutes,
      message: 'Outbox events not being published'
    });
  }
}

// Run health check every minute
setInterval(checkOutboxHealth, 60000);

Pitfall 3: CDC Replication Slot Filling Disk

Problem: Debezium connector goes down, PostgreSQL WAL accumulates.

Solution: Monitor replication slots and set retention limits:

-- Set WAL retention limit
ALTER SYSTEM SET wal_keep_size = '10GB';

-- Monitor slot status
SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as lag
FROM pg_replication_slots;

Alert if a slot is inactive for more than 5 minutes, indicating a publisher failure.

Comparison with Other Patterns

Outbox vs. Event Sourcing

Aspect	Outbox Pattern	Event Sourcing
Purpose	Reliable event publishing	Events as source of truth
Event Lifetime	Short-lived (deleted after publish)	Permanent append-only log
State Storage	Current state in tables	Derived from events
Complexity	Low	High
Query Model	Direct database queries	Requires projections/CQRS
Best For	E-commerce orders, workflows	Banking, audit systems

Key difference: In event sourcing, events are the permanent record. In outbox, events are a communication mechanism.

Outbox vs. Saga Pattern

The outbox pattern complements the saga pattern. Use outbox within each service participating in a saga:

// Order Service publishes OrderCreated via outbox
await db.transaction(async (trx) => {
  await trx('orders').insert(order);
  await trx('outbox').insert({ event_type: 'OrderCreated', payload: order });
});

// Saga Orchestrator receives OrderCreated, publishes commands via its own outbox
await db.transaction(async (trx) => {
  await trx('saga_state').insert({ saga_id: orderId, step: 'INVENTORY_PENDING' });
  await trx('outbox').insert({ event_type: 'ReserveInventory', payload: { orderId } });
});

Decision Framework

Polling is the starting point. The branches below say when to leave it:

Choose Polling when:

• Event volume < 1000/minute
• Getting started quickly
• Simple architecture preferred
• Database doesn’t support CDC

Choose CDC when:

• Event volume > 1000/minute
• Need < 1 second latency
• Production system at scale
• Already using Kafka

Choose DynamoDB + EventBridge when:

• Building on AWS
• Want serverless architecture
• Minimal operational overhead desired
• Cost-effective for moderate volumes

Production Readiness Checklist

Before deploying the outbox pattern to production:

• Cleanup strategy: Automated deletion of published events
• Monitoring: Outbox age, backlog size, publisher health
• Alerting: Lag exceeds threshold, publisher failures
• Idempotency: Inbox pattern or idempotency keys implemented
• Ordering: Partition key strategy for event ordering
• Dead Letter Queue: Failed events routed for investigation
• Schema versioning: Event payload versioning strategy
• Load testing: Verified at expected throughput
• Runbook: Documented recovery procedures
• Backup strategy: For outbox and inbox tables

Where the Default Holds

Polling holds until the outbox turns into a hot table or the poll interval becomes visible to users. From there the override goes in one of two directions: CDC when the volume justifies running Kafka Connect and watching replication slots, or DynamoDB Streams into an EventBridge Pipe when the data already lives in DynamoDB and you would rather write no relay code at all.

The relay you pick does not change the delivery guarantee. It is at-least-once in every case, which leaves consumer idempotency and outbox cleanup as the two pieces no implementation can skip.

References

• Microservices.io: Transactional Outbox Pattern - Canonical pattern definition by Chris Richardson
• AWS Prescriptive Guidance: Transactional Outbox Pattern - AWS architecture guidance for the outbox pattern
• Azure Architecture Center: Transactional Outbox with Cosmos DB - Microsoft reference implementation
• Debezium: Outbox Event Router - CDC-based outbox relay via Debezium SMT
• Event-Driven.io: Outbox and Inbox Patterns Explained - Delivery guarantees and at-least-once semantics
• Amazon EventBridge Pricing - List prices for Pipes requests and custom event bus ingestion
• Amazon DynamoDB On-Demand Pricing - Stream read request unit charges and the Lambda trigger exemption