What is the transactional outbox pattern and why is it needed?

Publishing to a broker and committing to a database are two separate systems. If the commit succeeds and the publish fails, downstream consumers never see the event. If the publish succeeds and the commit fails, consumers act on something that never happened. The outbox removes this gap.

BEGIN;

UPDATE orders
SET status = 'CONFIRMED'
WHERE id = 42 AND status = 'PENDING';

INSERT INTO outbox (id, aggregate_id, event_type, payload, created_at)
VALUES ('evt_01', 42, 'OrderConfirmed', '{"orderId":42}', now());

COMMIT;

How the relay works

• Polling publisher: a worker selects unpublished rows, publishes them, then marks them sent
• Change data capture: a log reader such as Debezium tails the database log and emits inserts
• The relay must tolerate publishing twice, because marking sent can fail after a successful publish
• Consumers must deduplicate by event id or make their effects idempotent
• Ordering is preserved only if the relay publishes in commit order for a given aggregate

The outbox table grows and needs cleanup. Delete or archive rows after they are confirmed sent, but keep them long enough for replay and audit. If the relay is down, events accumulate in the database, which is a visible backlog rather than silent loss. That is the main advantage over a best-effort publish.