Streaming has a silence problem.

The short version

When a Kafka partition goes quiet, is it done, or just slow? From the outside those look identical — and the answer decides whether you can safely act on what you've got. Netflix, Uber, and Walmart have each published how they handle it, and each one gives something up: a second pipeline, a lane of parallelism, correctness under replay. Sound engineering, every time. None of them asks Kafka the one thing Kafka already knows.

I hit this on an inventory stream for a Fortune 50 company: thirteen Kafka topics, three partitions each, hundreds of millions of events a day, every event for a given product required in strict event-time order — reading seven days of history from scratch and then going straight into live peak traffic. The first design put a thirty-minute window in front of everything, still needed a second job to sweep up what the window missed, and still let 5.5% of records out in the wrong order.

The version that shipped doesn't wait on a clock. It reads Kafka's offsets.

Here's how — and why the idea is older than Kafka.

The trade everyone makes / §01

This isn't one team's problem. Four strong engineering orgs, four sound trades:

• Walmart pins each item–store pair to a single partition, so only one consumer ever updates it. Ordering by pinning keys — it trades parallelism.
• Uber runs a separate backfill pipeline to recompute after late and out-of-order events, because the streaming path "can cause inaccuracies by dropping any events that arrive after watermarking." It trades a second pipeline.
• Netflix, backfilling Flink pipelines, found replayed history produced a different order and runaway state. It trades correctness and state.
• Confluent's own guidance for an idle Kafka partition: wait on it and stall, or ignore it and accept "many late messages once the partition becomes active again." It trades stalls or late data.

Every one is reasonable. The common thread is that each infers completeness from time — a window, a timeout, a wait — when the partition offsets say exactly where the data ends. That's the gap this closes.

Why the stream is scrambled / §02

A Kafka topic is split into partitions, and each partition is its own track. Inside a partition, order is guaranteed: record one, then record eleven, always. Across partitions, nothing coordinates anything — the consumers reading them are never at the same place at the same time. Kafka does exactly what it promises, the consumers do exactly what they promise, and the combined output is still scrambled for anything that cares about sequence.

You can't act on "sold" before "in stock" for the same unit. So before a pipeline releases anything, it has to answer one question: have I got everything up to this moment? Almost every answer in use today is a timer — the window, the idle timeout, the reconciliation job. Each one is a guess about silence.

An old fix: report, don't wait / §03

There's a precedent, and it's not a better clock. Early railways ran on time-interval working: a following train left once a fixed interval had passed, so the line "must" be clear.

In June 1889, a train stalled on a steep gradient outside Armagh, in Ireland. The rear carriages were uncoupled, ran back down the hill, and struck the train following behind. About eighty people died.

The following train had left Armagh once the required twenty minutes had passed. Twenty minutes had passed, so the line must be clear. Nobody at Armagh had any way to know it wasn't.

Britain didn't answer with a better watch. Parliament gave the Board of Trade power to require absolute block working on passenger railways: no train enters a section of track until the signal box ahead confirms the section is clear. Clearance stopped being a duration somebody waited out and became a fact somebody reported. A stalled train and a departed one no longer looked the same, because nobody was reading anything into silence anymore.

That's the entire move. Reading Kafka's offsets is the signal box.

The build — three pieces / §04

The principle is short: time moves only when the evidence proves it can. Making that real took three pieces, and the hard part wasn't where people expect.

One: the timer queue is the sort. Every record is parked in keyed state under its own event time, with an event-time timer for that exact millisecond. Nothing is emitted yet. When the watermark reaches that timestamp the timer fires and everything parked there leaves together. Flink fires a key's timers in ascending timestamp order, so we never sort anything. It's one method and a piece of state — the difficulty was never in here.

Two: the stream with no event time becomes the clock. One of the thirteen topics had no event time at all — it only published valid products for everything else to enrich with. So it had nothing to sort, and it could do what the others couldn't: emit watermarks from processing time, always the latest possible timestamp. The catch: a pipeline's clock is the minimum across its inputs, so a stream running at "now" changes nothing while the history streams are still being counted. I built this piece first and it had zero effect — it wasn't wrong, it was in the wrong order. The control stream only matters once the others leave the minimum, and a stream leaves the minimum when it's marked idle.

Three: Kafka decides when a stream is done — not a timeout. Flink's built-in idleness is a timer: no data for a while, mark it idle. But a timeout can't tell "starting up" from "finished," and startup is the worst case — connections form at different speeds, one topic sits quiet, the timeout fires, the others run the clock past event times it hasn't read yet, and its first record arrives already late. That's not a delay, it's a violation. So the timeout had to go. At startup, before reading a record, the job asks Kafka for the last offset in every partition and holds it. A topic is idle only once every partition has reached that number — not most, every one. No duration to tune, because the gate is a fact from Kafka.

// once, at job startup
Map<TopicPartition, Long> endOffsets = consumer.endOffsets(partitions);

// as records arrive
if (offset + 1 >= startupEndOffsets.get(partition)) markCaughtUp(partition);

What it bought / §05

The window held every record until it closed — thirty minutes just to read history in, again on every restart. Now a record is released as soon as the watermark passes it: on average under 100 milliseconds, including the records held back so they wouldn't come out of order. A record that arrives at minute one leaves at minute one. Ordering went the same way: two layers of timeout idleness let 5.5% of records out of order; one layer, 1.5%; idleness decided by Kafka's startup offsets, zero. The window is gone and so is the sweep job — one job reads seven days of history and then runs live.

What it doesn't promise / §06

Deterministic isn't clairvoyant. An event exists for the pipeline only once it lands in Kafka. If a producer holds a record back and it lands after the clock has already passed its event time, it goes out at once, out of turn, and the violation counter records it. No stream processor can order an event it can't see yet. Inside that boundary the guarantee is exact, and the watermark lag sets how wide the boundary is.

The same question, everywhere / §07

Once you've seen it, you see it in every system that decides what's true from streams that don't arrive together.

A river gauge goes quiet during a flood. Is the river steady, or did the water reach the sensor? In 2018 the USGS reported streamgages knocked offline by Hurricane Florence — the ones that mattered most. "No news" is the wrong thing to read into that silence.

A camera rig. Capture an athlete with several cameras and each is its own stream, like a partition; you can't fuse a 3D pose for an instant until every camera has reported it. A silent camera might have dropped a frame or be watching someone hold still. At i3d.ai it's the same question, dozens of times a second.

An agent. An AI agent calls a tool and hears nothing back. Agents are very good at treating silence as success — no error came back, so on to the next step at machine speed. An agent acting on a customer's account as it stood an hour ago is a train dispatched on a timer. The fix is the one the railways found: no step proceeds on silence; every step proceeds on a confirmation it can point to.

The cheapest reliability win in streaming isn't a faster engine or a bigger window. It's to stop inferring "done" from silence and ask the system that already knows. The window waits out the worst case for every record; the offset check waits exactly as long as the evidence requires, and not a millisecond more.

The stripped-down reference implementation — generator, handler, catch-up tracker — is the next thing I'm publishing.