Three ways to pay for time
Eve dares Surya to prove his default-to-the-truck advice, and he does it with a Tuesday, not a slogan. Each move from tomorrow-morning toward this-second, he says, buys a fresh reason to be awake, worth it only when a decision genuinely changes inside that window.
Asked to explain batch, micro-batch, and streaming slowly, Surya refuses the old-newer-newest timeline; they are three ways of paying for time. Batch is the nightly truck: one scheduled job over a day's tickets; a crash means resending it. Micro-batch is a shuttle van on a loop; Spark Structured Streaming is essentially this, and many self-described streamers are really on the shuttle. The belt is true event-at-a-time, Flink-style, milliseconds to seconds. The label matters less than state, windows, exactly-once, and backpressure, which is the belt outrunning the cook: a mature system slows, drops, or pages; a careless one loses the night.
A Tuesday at one in the morning
It begins at 1 a.m., after the cashier computer has closed its day and long before breakfast. Airflow wakes, not a person, and first fetches yesterday from an Oracle-class system, either by reading the diary of changes since the last bookmark or by a timestamp grab, which misses deletes and anyone who forgot to update the stamp. The diary wins; the stamp remains common.
Records arrive in the lake as Parquet files under yesterday's date. Picture one order, ID, merchant, amount, status, now a file instead of a row. Then a sensor, a small check Surya wants her to love: are the files present, in the promised quantity? If Monday's truck is late, the job waits or pages rather than publishing a zero.
Where the source is messy or huge, Spark washes it into a cleaner pile in the lake; tidy sources skip the sink. The warehouse then copies the files in as tables. This is bronze, still raw, though the order is now a queryable row. Keeping only pretty tables is a mistake, because future-you will disagree with present-you, and a raw shelf lets Tuesday be recooked without troubling Oracle.
Silver, gold, and a door that stays shut
dbt cooks inside the warehouse. Silver: amount becomes money rather than text, status a short allowed list, columns get names people actually use, each row means a single thing. Gold: the order is joined to customer, merchant, and calendar, facts with dimensions around them, so finance gets revenue by region without decoding Oracle's handwriting.
Tests run as part of the build: unique order ID, non-null amount, every order pointing at a real customer. A failure keeps the gold door shut and yesterday's good plate on the pass.
Surya: “Nobody eats the bad plate just because the clock moved.”
When it works, the door opens, caches refresh, Slack goes green; if not, retries, then a phone. Eve retells the whole thing in one line, and Surya says that sentence covers more than half the job listings.
The belt, at the same slow speed
Phones and their app emit taps, orders, and location pings onto a log (Kinesis, a managed Kafka, Event Hubs, Pub/Sub), written once and left for several cooks. A log rather than a call between services, because a call dies when the other side is away, and because pricing, fraud, and the lake archive can all read the same events. Order is kept per key, often user or merchant, across many parallel belts.
A schema registry bounces malformed events at the door into a dead-letter queue, a polite drawer opened later. A stream job, often the Spark shuttle, counts within a window such as orders per merchant per minute, waiting briefly for stragglers from tunnels before closing the book, because adding forever melts the cook. A checkpoint is the job's own note of how far it got; the log keeps a separate bookmark, the offset, which replay rewinds. Raw events also always land in the lake, since the log does not keep forever. The warehouse gets a drip for a near-live chart, but the hottest readers take the stream directly; the dashboard is rarely the fastest mouth.
One decision, and the test to steal
Eve wants one decision where the belt pays for itself. Surya's is checkout dying: pay-button errors jump, and waiting until coffee means a morning of no sales. A windowed count sees it in seconds and pages someone. Yesterday's sales by region can run twenty minutes late without harm; wrong, they cost a quarter, so they stay on the truck. The test to steal: would a person act differently within this minute if the number were late? If not, the stream is vanity.
Failures differ. The truck reruns the day; the shuttle resumes from its checkpoint, losing minutes; the belt replays from the log's bookmark and hopes its state logic is not poetry. Each hop multiplies operations, from rotting checkpoints on the shuttle to state, ordering, backpressure, and afternoon pages on the belt. Cost follows: big scans are cheap per row; always-on stream compute never sleeps.
Do the numbers even agree? Nightly gold can reconcile against the farm; streams are eventually consistent and duplicate unless exactly-once was paid for, and sometimes even then. At-least-once is the honest default, because networks retry and phones double-tap, so the write is built so that twice equals once: a merge on a unique event ID.
Surya: “If you cannot merge twice, you are not ready to stream.”
The default, phrased for stealing: truck, then shuttle, then belt, skipping ahead only when a decision that changes inside the window can be named aloud. Otherwise you bought a personality, and Surya calls that the field's most expensive purchase.
