Log-Based Messaging & Streaming

A retained log changes the central question of messaging. A queue asks “which worker owns this message now?” A log asks “which immutable records exist, and where is each reader?” Reading does not normally delete a record. Retention or compaction changes the stored history independently of any one reader.

That separation enables replay, independent projections, catch-up after downtime, and stream processing. It also turns the partition key, retention policy, consumer position, and replication acknowledgement into parts of the data model. They cannot be chosen as afterthoughts.

One topic is many ordered logs #

A topic is split into partitions. Each partition is an append-only ordered sequence; every record receives a monotonically increasing offset within that partition. There is no meaningful topic-wide offset and no default total order across partitions.

The partition key simultaneously chooses:

• the ordering boundary — equal keys normally land in the same partition;
• the unit of parallel consumption within a consumer group;
• the unit of replication and leader ownership;
• the hotspot distribution;
• the migration problem when partition count or partitioning logic changes.

For orders, order_id is often a sound key because all state transitions for one order must be observed in order while unrelated orders can proceed in parallel. customer_id gives a wider order boundary but risks hot customers. A constant key gives total order at the cost of one partition's throughput and one active consumer per group.

Append, fetch, and offset are separate operations #

Producers append records to the current leader of a partition. Consumers fetch batches starting at an offset. A consumer's current processing position and its last committed restart position are different states:

1. fetch records 120–169;
2. process through 145;
3. commit offset 146, meaning the next restart should begin at 146;
4. continue processing in memory.

If the process crashes after applying record 145 but before committing 146, it reprocesses records from the older committed position. If it commits first and crashes before applying, it skips work. That is the same irreducible failure ambiguity seen with queue acknowledgements; offsets merely expose the position explicitly.

Offset commits are checkpoints, not proofs that arbitrary side effects happened. A correct consumer chooses the coupling:

For an external database, common solutions are an idempotency/inbox table in the database transaction or an outbox that publishes the database commit as a separate durable step. See Idempotency & Exactly-Once Effects.

Consumer groups turn partitions into work slots #

A consumer group is one logical subscriber. At a given moment each partition is assigned to at most one active member of that group, so members divide the partitions while preserving per-partition order. A second group reads the same records independently with its own positions.

Consequences follow directly:

• with 24 partitions, a group can use at most 24 active partition owners;
• adding a 25th consumer does not add read parallelism unless a consumer can cooperatively share work after fetching;
• one slow partition can dominate the group's completion time even when other consumers are idle;
• consumer membership changes require a rebalance that revokes and reassigns partitions;
• work in flight during revocation needs a defined drain, cancellation, and offset-commit protocol.

Rebalancing is a correctness event, not only a latency spike. If the old owner continues writing after the new owner starts, effects can overlap. Generation or epoch checks, cooperative assignment, bounded processing time, and idempotent effects contain that race.

Retention makes replay possible — within a window #

In a queue, successful acknowledgement normally makes the queued copy eligible for deletion. In a log, retention is independent of a specific read. A reader can reset its position and rebuild a cache, index, materialized view, or model from the retained history.

The simplest storage estimate is

$$B_{stored} \approx \frac{R_{ingress}\,T_{retention}\,F_{replication}}{C_{compression}},$$

where $R_{ingress}$ is uncompressed bytes per second, $T_{retention}$ is seconds retained, $F_{replication}$ is replica count, and $C_{compression}$ is the achieved compression ratio. Add index, segment, filesystem, headroom, and rebalance/recovery traffic rather than sizing to the estimate exactly.

Retention can be time-based, size-based, or both. A consumer whose lag grows beyond the retained window has not merely become slow; it has lost the ability to reconstruct the missing interval from this log. Alert on time lag to the retention boundary, not only record count.

Replay also replays defects and side effects. A safe replay plan identifies:

• the destination as a new projection, a truncated projection, or an idempotently updated existing one;
• the code and schema version used to interpret old records;
• how emails, payments, webhooks, and other irreversible effects are disabled or deduplicated;
• how replay traffic is rate-limited so it does not starve the live tail;
• the cutover point at which rebuilt state becomes authoritative.

Compaction is not deduplication #

Kafka log compaction retains at least the latest known record for each key in a partition while preserving the offsets and order of records that remain. A record with a key and null value acts as a tombstone; delete-marker retention must be long enough for rebuilding consumers to observe it.

Compaction is useful for change streams such as “latest customer profile by customer id.” It does not mean a reader sees only one record per key, does not remove duplicates at ingest, and does not provide a full audit history. The uncompacted head can contain many versions; the cleaner runs asynchronously; and tombstones eventually disappear. Use time/size retention for event history, compaction for reconstructable keyed state, or both when the semantics are explicit.

Replication: accepted, committed, durable, and visible #

For a replicated partition, the leader orders appends and followers copy that order. Kafka considers a record committed after all replicas currently in the in-sync replica set have applied it. Producer acknowledgement configuration, minimum in-sync replica policy, replica health, and unclean leader-election policy together determine whether availability or acknowledged-data safety wins during failure.

The vocabulary must stay precise:

• sent — the producer attempted a request;
• acknowledged — the configured broker condition was met;
• committed — the platform's replication visibility condition was met;
• durable under a named failure model — enough independent copies survive the failures being considered;
• processed — a consumer performed its intended effect.

None implies the next. A client timeout can leave an append outcome ambiguous, so retries may create duplicates unless the producer protocol deduplicates them. An acknowledged record can still be lost if the selected acknowledgement and election policies allow a replica without it to become leader.

Exactly-once has a perimeter #

Kafka's idempotent producer prevents duplicate log appends caused by producer retries within its protocol session rules. Kafka transactions can atomically publish to multiple Kafka partitions and commit consumed offsets. A consumer using read-committed isolation can avoid records from aborted transactions. Together, these provide exactly-once processing for a read → process → write pipeline whose inputs, outputs, and positions all live in Kafka; Kafka Streams uses this boundary.

They do not atomically include an arbitrary PostgreSQL transaction, Stripe charge, email, filesystem write, or another broker. Describing such a workflow as “Kafka exactly-once” hides the only boundary that matters. Cross-system effects still require idempotency, an outbox/inbox, or a coordinator with an honest failure model.

Event time, processing time, and late data #

Streaming systems expose at least two clocks:

• event time: when the domain event occurred, carried in the record;
• processing time: when this consumer observed and processed it.

Network delay, mobile clients, retries, and replays make them diverge. Windowed aggregations therefore need a lateness policy and a watermark — a claim about how far event time is believed to have progressed. A watermark is not a proof that no older event will arrive. It determines when results are emitted, when state can be reclaimed, and whether late records update, retract, or go to a side output.

The log supplies ordered records per partition; the stream processor supplies event-time windows, state stores, joins, checkpoints, and result-update semantics. Do not attribute processor guarantees to the broker itself.

Backpressure appears as lag #

Queue systems expose waiting messages and unacknowledged deliveries. Log systems expose a distance between a partition's log end and a group's position. For partition $p$,

$$L_p=O_{end,p}-O_{committed,p}.$$

Record lag is only a workload proxy: one record may cost microseconds and the next minutes. Track lag in records and bytes, oldest unprocessed event time, consume rate, process latency, rebalance time, and remaining retention window.

If arrival rate remains above service rate, adding consumers helps only until every partition has an owner. Beyond that point the options are to make each consumer faster, increase partitions with an explicit key migration plan, split the workload into another group or topic, shed optional work, or reduce arrival rate. The queueing math is covered in Backpressure & Load Shedding.

Queue or retained log? #

Many systems need both. An order service can publish facts to a retained log for audit and projections, while a routing processor derives short-lived, retryable fulfillment commands into work queues. The bridge must preserve ids, schema versions, causation, and idempotency; otherwise it merely relocates the ambiguity.

Systems that expose log-like semantics #

NATS JetStream illustrates why labels alone are insufficient: a stream can retain by limits like a log, by consumer interest, or by work-queue policy. RabbitMQ has both quorum queues and streams. Pulsar offers exclusive, shared, and failover subscriptions over retained storage. Compare the configured state machine, not whether a product calls itself a broker, queue, or streaming platform.

Worked design: rebuildable account balances #

Assume each accepted ledger entry is immutable and every account's entries must be applied in order.

1. Use account_id as the partition key; each account has one ordered lane.
2. The producer includes entry_id, schema version, effective event time, causation id, and the source transaction id.
3. The balance projection stores (account_id, balance, last_applied_entry) in one database transaction and rejects an already applied entry_id.
4. The group commits offsets only after those transactions commit.
5. A new projection version writes to new tables and replays from the earliest required offset under a bounded traffic budget.
6. Cutover compares invariants — entry count, sums, last ids, and sampled balances — before switching reads.

This design tolerates redelivery and enables rebuilds. It does not make the log the accounting authority by magic: acceptance rules, double-entry invariants, reversal semantics, and source transaction durability still belong to the domain system.

Design review checklist #

• What immutable fact does each record represent, and may it ever be corrected or deleted?
• Which key defines the required order, and how skewed is its distribution?
• What happens to key ordering if partition count or partitioner changes?
• What is the replication acknowledgement condition, and which failures may lose acknowledged data?
• Who commits positions, when, and with which side effects atomically coupled?
• How are duplicates and poisoned records detected across restarts and replay?
• How long is history retained, and how early will operators learn that a group is approaching the retention boundary?
• Is compaction preserving state, or is an audit history required?
• What happens during rebalance to in-flight work and stale owners?
• How are schemas interpreted by consumers months behind the head?
• Can a replay safely suppress or deduplicate external side effects?
• Which exact systems participate in any “exactly-once” claim?

If the partition key, offset transaction, or replay plan is unknown, the log is not yet a reliable architecture — it is only storage with hopeful consumers.