Hope, scheduled, is not DataOps
Eve describes her old team: strong models, and a Slack channel asking why Tuesday was empty. Surya calls that hope on a timer, not DataOps, which is not something you purchase but running the kitchen the way software people already run the customer-facing app: branches, reviews, tests, and a staging room genuinely separate from the dining room.
Nobody edits the live recipe at the pass. You branch, change the SQL, open a pull request, a second human reads it, tests run before the merge. Shipping a metric change unreviewed puts you in a garage, not a bank; know which you are. The tests are concrete: keys may not be null, ticket numbers must be unique, each ticket has to reference a customer that exists. dbt normalized this for SQL; a failing test keeps the plate in the kitchen.
Staging is a room, not a hat
Dev, staging and prod each get their own credentials and database copies, and Surya is careful to say databases, not the rented compute Snowflake calls a virtual warehouse. A real staging room holds production-shaped data or a clone, without production credentials and without serving the chief executive; prod with the word staging in the name is a hat.
Code review for SQL sounds fussy until a join doubles revenue. The failure that converted him: one customer joined to many addresses with no current filter, freshness green, two cows stapled together. A uniqueness test at the grain would have failed in staging; an executive found it in a meeting instead. Grown-up deploy: build the new table beside the old, check, swap only on a pass.
The schedule board lives in git
Eve still pictures a ceiling fan when Airflow comes up. Surya reframes it as the schedule board. Jobs are written in Python; the job graphs, the dags, show what runs, in what sequence, and what happens when the truck is late.
The graphs belong in the same git as the recipes, not in a browser someone clicked through late at night, because at three in the morning you will ask who reordered the run. The board does not chop onions: it tells Spark, dbt and the warehouse when to start. Letting Airflow crunch the food itself is a new way to be down.
Did the right cow arrive
Eve's team once had freshness green while the chief executive's number was wrong. Surya separates the ideas: freshness asks whether the truck arrived; observability asks whether the right cow arrived and still resembles one. It watches volume against last Monday, sudden nulls, drift, duration and cost per run.
Surya: “Tests assert what you already fear. Observability is how you notice the cow is a horse.”
Drift is valid but wrong: every check green while ticket size slides after a join silently drops one payment type. He refuses a vendor shopping list: begin with the warehouse already on the bill, since Redshift, Snowflake, BigQuery and Synapse all support tests and keep query logs. Lineage lets you walk upstream from chart to bronze in minutes, and downstream to see who else eats from a table you might break.
Not seven. When the food is there
Eve assumes jobs run at seven because somebody liked the number. That is the clock-aware board. Data-aware scheduling makes the dashboard job wait for yesterday's orders to exist and pass their tests; firing the cooks at seven with a late truck yields an empty plate and a green check, which is Eve's Slack channel exactly. Deadlines still matter, kept by noticing at four that the truck is late. If tests pass but the number is still wrong, that is a test gap for observability, not a scheduling failure.
On the pitch that pipelines are over, Surya says zero-ETL means either a managed copy under the table, such as Amazon's from its database into Redshift, or query federation. Fine for a prototype; pulling three years of sales across the building is the old lunch-rush problem in new clothes. Somebody still models the tickets, writes revenue once and masks exit doors, and without a landed bronze crate you cannot rebuild or show an auditor last March. Change data capture, merging the diary of inserts, updates and deletes, remains the honest way to follow a database; Debezium is the open-source name.
Surya: “The vendor retired a hose. They did not retire the kitchen.”
The bill is something you designed
Eve's finance partner suspects the team burns money for sport. Surya calls the bill an architectural output, designed but not on purpose. Object storage, S3, Blob or Cloud Storage, costs pennies unless you abuse it with requests or tiny files. A stream bills you for staying switched on; a warehouse bills you for compute left on, and Eve makes him restate the two rooms: the pantry is cheap; the Snowflake-style virtual warehouse is a cook rented by the minute, dismissed when the line is quiet. Auto-suspend is not optional.
Also: partition so yesterday's question skips 2019, cook incrementally, separate finance from the intern's cluster, and tag everything to a team so cost lands on whoever caused it. The best trick is deletion; tables unqueried for months are a running gas bill, and he will not invent a percentage.
His closing map: contracts stop bad food, observability spots it, DataOps prevents serving it twice, scheduling prevents the empty plate, FinOps notices the ocean being boiled for tea. Skip one and the rest are theater. The Slack channel retires when an empty Tuesday means the truck really never came, and everyone already knew.
