Stand in each room and know its purpose
Eve can retell the fear in ride apps, living rooms, hospitals, and banks, but not the counter itself. She asks Surya to walk the line.
Surya agrees, with a rule laid down before any vendor is named: a station's purpose outranks the logo on its door. He names Amazon, Microsoft, and Google doors only because Eve asked for them.
Last time, Surya says, he gestured at rooms. This time Eve should be able to enter any one and know why it exists, so that when a salesperson waves a brand she can answer: a fridge, a cook, or a schedule board impersonating a cook. The job is still the copy.
Where the food is born and how it leaves
The farm is the cashier computers: Oracle, SQL Server, Postgres. Row stores built to record a sale this instant and keep it. Customer conversations often sit in Salesforce or a relative, reachable only through an API that can rate-limit you or grow a custom field one Thursday with no announcement.
The copy comes in two flavors. The blunt one is a nightly full snapshot: photocopying the whole library. It burdens the cashier machine, takes hours, and shows only today's shelf. The better one follows the diary. Every database records what it inserted, updated, and deleted; change data capture, shortened to CDC, reads that record and moves only the changes. Debezium is the open-source name; the cloud doors are Database Migration Service, Datastream, and Microsoft's own change-feed products.
Two decades of Informatica? The old truck keeps hauling legacy feeds while new work lands in the lake or on a belt; many real platforms are hybrids.
The dock, the walk-in, and the old district
Ingest is the loading dock: a scheduled truck carrying hours of tickets that can be sent back, or a belt, meaning an ordered log written once and read by many cooks. Doors, named once: Kinesis or a managed Kafka on Amazon, Event Hubs on Microsoft (it can even talk Kafka), Pub/Sub on Google.
Then the food needs a cheap place to sit: object storage, a walk-in for files rather than rows. S3, Blob Storage, and Cloud Storage are the same fridge under three logos. Most of it lands raw because nobody yet knows how Tuesday-you will cook it. Files are often Parquet, columnar, so three wanted columns need not drag thirty-seven others through the kitchen. Shelves are cut by date so a cook wanting yesterday need not open the whole fridge. Structure arrives on read, not on write: a forgiving landing zone, a bad thing to hand finance with a fork. Hadoop was the old warehouse district; the cloud swapped in object storage and Spark and changed the sign.
Two cooks and a phone book
Apache Spark washes the ugly, huge piles, one engine on every cloud: EMR or Glue on Amazon, Dataproc on Google, Synapse Spark and now Fabric on Microsoft. Glue also sells a phone book. A catalog records which datasets exist, their shape, and their shelf: Purview on Microsoft, Dataplex on Google. Without one, Surya says, new cooks fear the fridge and tables get copied twice under different names.
Surya: “Discovery is not a luxury. It's how the pile stays a kitchen.”
The lock is least privilege: the role that lands raw files cannot open the locked shelf.
The pantry is the warehouse: organized, typed, SQL-first, and priced for those manners. Redshift, BigQuery, Synapse and now Fabric, and Snowflake on any of the three. Decoupled storage and compute, Eve guesses, just means fridge and cook are billed separately; Surya confirms it. Again: a Snowflake virtual warehouse is compute, the cook and not the shelves.
Eve notices he has hired two cooks, which Surya calls the right question. Spark preps the dirty, many-file arrivals; the warehouse handles the final mile of meaning, with clean types, readable joins, and figures finance will stand behind. Merge them and you get a lake called self-service, or sludge in the pantry and a bill like a second rent.
Recipes in writing and a board that yells
dbt keeps recipes as plain SQL selects in git, reviewed like code, and builds them as tables and views in dependency order with tests and documentation. Here the letters flip: today's habit loads raw first and transforms inside the warehouse, which is why a raw shelf exists for the day present-you turns out to be wrong. The older order cooked in a side kitchen and shipped only dessert to a costly pantry, so a changed recipe meant phoning that kitchen, or the farm.
Airflow is the board on the wall. It chops nothing. It dispatches the truck, waits for files to land, cues Spark, the warehouse, dbt, and the tests on the gold door, and pages someone on failure, all in Python. Managed boards: Amazon Managed Workflows for Apache Airflow, Composer at Google, and usually Data Factory, Synapse pipelines, or now Fabric pipelines at Microsoft. Keep the board away from the knives: an orchestrator that also lifts is a fresh way to be down.
Surya: “Most of this job is obvious once you say it. The hard part is saying it while a vendor is talking.”
The line without brands: sources, a copy path, a cheap fridge, one cook for dirty work, a labeled pantry, written recipes, a board that shouts. Know what each station is for, and no one can sell you a blender as a walk-in.
