S1 · Episode 11

Data Governance: Owners, Catalogs, and Lineage

Runtime 14:07 AI-narrated

About this episode

Someone shipped a dashboard. Nobody knows who owns the table. Do not start with a tool. Start with a name. If nobody gets the page, you do not have governance. You have folklore.

Eve and Surya teach the phone book (Glue Data Catalog, Microsoft Purview, Google Dataplex) and why it is useless if the owner field is empty. Lineage means walking a wrong chart back to the join that blew up. The hard security problem is people with legitimate access doing illegitimate curiosity. Master data is one customer, three spellings, and a steward. Retention is long enough for the lawyer and short enough for the privacy law. Time travel is cute until you cannot delete a person. A small council writes a few global rules. The platform enforces them in automated build checks, not in a meeting everyone skips.

A name before a tool

Eve opens with a scene: a dashboard has shipped and nobody can say who owns the table underneath it. Surya's first move is a person, not software. Without someone who is paged when the number is wrong, the company runs on rumor: maybe Maya owned it once, but Maya left in March, and meanwhile two teams bring two revenues to one meeting and blame the kitchen.

The deeper gap is not who cooked the plate but who is permitted to call it wrong. Governance, in his words, is dull work, and the contact card is its first brick. An owner is the person entitled to declare a table true, stale, or unfit for that dashboard, never “the platform team” or an unread mailing list. And she begins with ten tables, not two thousand.

The phone book with empty pages

Eve keeps being sold a catalog. Surya reduces it to a phone book: what is out there, whose it is, can it be trusted. Glue Data Catalog on Amazon, Purview on Microsoft, Dataplex on Google: one book, three covers.

A crawler can infer columns but not who carries the pager or whether a table belongs in gold. With the owner field empty, an analyst hunting for customer lifetime value finds half a dozen near-misses, takes the friendliest name, and builds another.

A useful page carries a human-typeable name, a sentence saying what one row is, an owner, a freshness promise, a certified-or-leftover flag, and a danger tag. The owner fills that in, not the crawler; mature teams refuse gold to anything missing those fields.

Walking a wrong chart back to the farm

Table found, chart still wrong: now Eve needs lineage, which Surya defines as a graph she can click while the executive is in the room.

His slow Tuesday: a revenue chart has holes in customer segment, and you walk from the chart to the gold revenue table, to the customer dimension it joins, down through silver and bronze to the field the sales tool should have delivered overnight. Somewhere a join collapsed, perhaps after a deploy changed a key. He is describing a shape, not a real company; the payoff is a broken join named in minutes.

The graph is stitched from the recipes (dbt records which model feeds which), the schedule board, the catalog, and the dashboard tool. He names no vendor but sets a test: a graph you cannot walk is a demo. Read downstream before dropping a column, it is a warning rather than an autopsy.

Locks, fences, and legitimate curiosity

Eve wants security without a sermon. Surya offers least privilege: the smallest kitchen key that still lets someone cook, granted through roles, not personal hero accounts. A marketing role reads gold and stays out of bronze, the crate of raw names and emails. Row-level fences applied at query time let a hospital analyst see one site from a single table. Emails, phones, and government IDs are masked; the fields that would make headlines are tokenized, with the mapping in a locked box, so joins still work.

Encryption at rest and on the wire is table stakes (S3, Blob Storage, and Cloud Storage already do it), but that is a lock on the door, not a list of who holds keys. The real difficulty is people with legitimate badges looking up an ex or a journalist. Query logs on sensitive tables catch it; reviewing who holds sensitive roles prevents it.

Three spellings, a steward, and the fridge

Eve glazes over at “master data,” so Surya starts with the disease, inventing a company: billing has Acme, sales has Acme Corporation, support has Acme (U.S.), so every gold join on customer becomes an argument. Confident matches merge; fuzzy ones go to a steward, a named business person who rules on which spelling and which source wins each field. That is politics, so it sits in governance with a neck attached. In a hospital a false merge fuses two patients; at a bank a missed match is a compliance matter.

Retention pulls both ways: keep records long enough for the lawyer yet delete forgotten people for the privacy law. He refuses to invent a number of years. Write the rule down and let object-store lifecycle enforce it: hot, cheaper, gone, unless a legal hold says stay. Warehouse time travel is a nightmare if yesterday's snapshot keeps serving a person who asked to vanish.

Four posters, a paved road, and Monday

Eve cannot memorize GDPR, HIPAA, Sarbanes-Oxley, and the payment card rules. Surya says memorize the question: which identifiers may not enter analytics, which access must be logged, which number must be replayable. Know your data is the phone book; trace it is lineage; control it is least privilege and masking; prove it is the logs.

The operating model need not be a meeting. A platform team paves a road so registering a table and getting scoped access takes minutes, or people trade spreadsheets instead. Domains own their gold. A small council writes a few global rules on personal data, naming, and interoperability, enforced in automated build checks that stop an unowned table from shipping.

Surya: “A policy that requires humans to remember it is a policy that will be violated on a Thursday.”

Monday's task is the ten tables the business really uses: an owner and a freshness promise on each, then lineage. The rest can wait.

Surya: “Governance that starts with everything starts with nothing.”

Takeaway

Monday: put an owner and a freshness promise on the ten tables the business actually uses.