S1 · Episode 12

How Generative AI Changes Data Engineering

Runtime 13:16 AI-narrated

About this episode

Does this job still exist, or do we all become prompt people? The job gets weirder, not smaller. Models eat data. They also emit data. Someone still has to move it without breaking it, losing it, or leaking it, including leaking it into a prompt.

Eve and Surya recap the season, then get specific: drafting SQL and tests gets faster; judgment does not. Confident-wrong is the new bad data. A golden set is a folder of the ugliest cases you have ever seen, run on every change. Lineage now includes prompts, retrieved chunks, and tool calls. Secrets in a prompt are the new password checked into the code repo. The old gates still apply: mask first, and a human before money or medical.

Drafting gets cheaper, judgment does not

Eve arrives at the final episode with one question left for her neighbor's sake: is this still a job, or does everyone become a prompt person? Surya says the work turns stranger, not smaller. A model consumes and produces data, and somebody still has to carry it around without breaking, losing, or leaking it, prompts included. The kitchen stays; it simply seats one more diner who never tires.

What speeds up is the typing: a first draft of SQL, of a test, of an explanation for why a DAG failed at six in the morning. Staring at a failed run was never the craft. What does not speed up is deciding which table is true, which number may enter a prompt, and whether an agent may write anything back.

Surya: “Pipeline authorship is getting cheaper. Pipeline judgment is not.”

A hand on the stove

Eve asks about write-back. Reading a table under a role is fine, Surya says; writing one, sending an email, or issuing a refund puts a hand on the stove. Let the agent draft; a human clicks.

Merging a pull request because it sounded confident, he adds, is hiring a junior who never sleeps and is never embarrassed. The skill that appreciates is telling gold from leftover soup.

Confident wrong, and a folder of ugly cases

Eve recalls the bronze, silver, and gold gates. Keep them, Surya says, and add a new category of junk. A hallucinated join reads like a senior engineer under deadline pressure: fluent, sure of itself, and made up. A wrong figure wrapped in a polished paragraph is camouflage. His Tuesday: a plain-English request for revenue by segment, a query against a near-gold table, and a two-decimal answer nobody questions because the prose around it sounds informed.

The remedy is evals, held to the standard of dbt tests: nothing ships because it sounds right. The golden set is a folder of the worst cases you have ever seen, replayed whenever anyone or any model touches the recipes: the query that hit the deprecated table, the prompt that nearly leaked an email, the double-counting join. A set made only of happy questions is a costume. Lineage now covers the prompt, the retrieved chunks, the tool called, and the role the agent wore. If you cannot replay why it said that, you built a demo; same walk as last episode, longer hallway.

Old gates, new surface

Does governance need new rules? Surya calls it old gates on a new surface. Pasting a key, a token, or a customer record into a chat window to get the SQL written is a leak by another route: the warehouse saw no door open, but the chat log recorded everything. Prompts deserve the discipline of a code repo. Since people paste anyway, the paved road has to redact, masking before retrieval. The agent counts as a user, so it gets a role, a minimal key, and a log, and it does not wander bronze because the prompt sounded official. Where money or medical care is involved, a model may draft the email or the refund, but a human approves.

Forgetting a person gets harder. The delete must reach the extra fridge, the index of document chunks turned into searchable numbers, or a fired employee's files stay retrievable tomorrow. Classify documents on landing so restricted pages never enter a company-wide index. Contracts, tickets, call notes, and PDFs are landed raw, extracted, and gated: new payloads, same shelves.

Copilots, catalogs, and the phone book as lunch

Vendors will claim to have solved this with copilots for the catalog, chatbots for the lake, and agents that supposedly run Airflow. Some of it helps, Surya allows, but bronze, contracts, and the owner field do not go away. He crowns none of Amazon, Microsoft, Google, or the warehouse brands; all are racing to put a model beside your data, and the real moat is the previous eleven episodes: context, permissions, audit trail.

Aim a model at ten thousand undescribed tables and it is confidently wrong at scale; aim it at owned, certified gold and it stands a chance. Monday's ten tables keep the copilot honest; the catalog work has become the prompt, which is every reason it finally gets budget.

The kitchen, the spine, and one sentence

Eve asks for the job restated with a model in the room. Data still moves from where it is created to where it is useful, which may now be a prompt. Broken is a wrong number inside a confident sentence; lost is a document that never reached the index; leaked is a secret in a prompt or an agent emailing a chart it should never have seen. Her neighbor keeps her job, approves more, types less, and still owns the not-breaking; Eve was looking at lunch while her neighbor named the stations.

The series in a walk: a kitchen the diner never sees; a copy out of the system where data is born; a deliberate choice between late and wrong; a stack of stations, not a shopping list; the truck by default; lakehouse manners on cheap storage; contracts, nameable gold, quality gates, a named human on every table, and then, maybe, a model. Tools rotate; the fear and the copy job do not. The sentence Eve can carry home is the one the series began with, about moving data from where it is created to where it is useful, intact.

Surya: “If a model helps you do that, keep it. If it only helps you sound busy, fire it.”

Eve says she will repeat that to her neighbor.

Takeaway

Takeaway: tell your neighbour you finally see the kitchen. The plate arriving on time is not an accident.