Gates between rooms, not a crew at the end
Eve opens with the poster line, bad data is worse than no data, and asks for the engineering. Surya's answer is gates: food does not advance to the next station until a person or a check signs for it. A late-night cleanup crew arrives too late; inspect only at the pass and everyone who took from an earlier shelf has already eaten the mistake.
Bronze, silver and gold describe the amount of work done to the food, not the prestige of the logo. Bronze is the delivery still in its crate, untouched and append-only. Why keep the crate? Because your future self will disagree with your present self, a regulator will want to know what arrived, and one day the wash job will break. Corrections get appended, never edited in.
Surya: “The crate is a diary, not a whiteboard.”
When the wash does break, silver and gold are garbage, and you rerun a fixed wash over the week's crates. Bronze is boring on purpose.
Three shelves and a signature on each door
Silver is washed and labeled: types correct, keys present, duplicates gone, odd rows quarantined with a note, one shared customer ID. Gold is plated for the business, the dining room from the previous episode.
Eve suggests promotion is a test, not a vibe, and Surya agrees. Fail the test and the data stays on the earlier shelf. A good failure halts at the door, leaves gold on Monday, and pages a human; the bad one is gold refreshing anyway because someone tired of red boxes. Location is furniture: bronze in cheap object storage such as S3, Blob or Cloud Storage; silver and gold usually in a queryable warehouse.
A bouncer and a labeled drawer on the belt
A live belt cannot pause for one ticket, so bad events are refused at the door of the log by a schema registry that forces the producer to match the contract. Amazon offers Glue Schema Registry, Confluent's is common on Kafka; Surya picks no church. The bouncer turns away a missing field, a price that is a word, a rename that would break forty downstream cooks; incompatible change dies at registration rather than in someone else's Airflow job at night.
Events that still slip through go to a dead letter, a labeled junk drawer on a side belt with the error attached. The processor neither crashes, which would stop the good tickets, nor hides the evidence. A healthy drawer trickles; one abruptly swallowing a noticeable share of traffic is a siren, and often the first sign of someone's bad deploy. Then you fix the producer and replay rather than pretending those orders never existed. Each event carries an ID, so seeing it twice counts once.
Surya: “Duplicates are not a moral failing. Unhandled duplicates are.”
The nightly crate gets washed
For the truck, once a file lands in bronze: schema against the contract, where a column that vanished, appeared or changed type either fails the load or evolves deliberately. Nulls: order ID, customer ID, amount and timestamp may not be empty; critical checks block, softer ones warn.
Duplicates come from replayed change data capture and restarted extracts, so silver keeps the latest row per business key deterministically and proves it with a unique test. Volume is compared with your own recent Mondays, not a figure invented on the show; too little is a partial extract, several Mondays' worth a duplicated feed. Freshness is the silent one: if yesterday never arrives nothing errors and the dashboard calmly resembles last week. It is a promise about when yesterday appears; the clock may measure that promise, but it must not start the dessert. Rejected rows go to an error table. dbt tests assert; Airflow refuses to promote when they fail.
Gold asks whether the number is true
Silver checked shape; gold checks truth. Surya's method is to sum the warehouse and sum the cashier computer for yesterday, cheaply, and compare, to the penny where money is involved. Eve objects that an earlier episode said never to query the register. He asks her to hold both: no three years by region at lunch, yes to a cheap aggregate at a quiet hour along a separate path, so your own pipe is not marking its own homework.
Missing pennies mean dropped rows, double counting or a multiplying join, so money gets no tolerance; other measures may. If the sums differ you slice by region and hour, trace suspect keys back to bronze, and log it, since auditors prefer a film to a feeling. Business rules also run as tests: nothing delivered before it is created, refunds never above the original. Drift is the same ghost as before, right shape, wrong number; it says look here, because sometimes a storm or a closed region really did change the business.
Common mistakes, and the same gates everywhere
Eve keeps waiting for the cleanup crew; Surya says none works, and gates are cheaper than apologies. Usual mistakes: fifty tests on gold and none at the door, a dead letter used as a trash can, gold reconciled against gold, and a human overriding the gate for an impatient chief executive, which is how the crew gets hired and never leaves. The human wrote the gate and gets the page; tasting every plate on a Friday night is not a system.
The gates repeat across clouds: a schema registry at the stream door on managed Kafka, Event Hubs or Pub/Sub; a dead-letter topic, built in on Pub/Sub, hand-built elsewhere; warehouse tests; an object store for the crate. Nothing moves until a test signs. And when the drawer suddenly fills, that is usually not a data problem first but a bad deploy, waving.
