Use cases

AI & data use cases

Twelve patterns we implement in production. Each one is drawn with the control that makes it safe to run — the gate, the threshold, the person who approves — because that part is the actual engineering, and it is what a demo leaves out.

Records from files, APIs and change-data-capture pass through a validation gate that routes them to load unchanged, normalise then load, or quarantine for human review.
Pattern 01

Data cleaning and preprocessing

  • Anomaly and gap detection: Rules plus ML and LLM classifiers flag null spikes, outliers and schema drift before a load lands in Snowflake or Databricks — not after a dashboard goes wrong.
  • Text normalization: Abbreviations, locales and free-text synonyms standardised against an agreed dictionary, with human review for anything the rules are not confident about.
  • Code mapping: Supplier and vendor codes aligned to internal MDM keys — deterministic maps first, fuzzy and semantic matching only for what is left over.
A classifier reads column names, sample values and existing documentation, applies confident tags automatically and sends the rest to a steward queue.
Pattern 02

Data cataloging and automated tagging

  • Semantic tagging: Column content — not just column names — classified with embeddings and rules, so notes_free and cust_id are not treated the same way.
  • Metadata extraction: Structural and operational metadata pulled from PDFs, object storage and legacy database schemas into one catalog.
  • Tags that do something: A tag is not documentation: applied tags are what the masking policies in the governance pattern below actually bind to.
A question passes through a semantic layer of certified metrics and allow-listed models before SQL is generated; requests for raw PII or cross-tenant joins are refused, and every prompt is logged.
Pattern 03

Natural language querying and semantic access

  • Natural language to SQL: Questions routed through a governed semantic layer — dbt metrics and semantic models — so generated SQL can only touch approved tables and approved metric definitions.
  • Refusal before generation: Cross-tenant joins and unmasked PII columns are blocked by policy before the model is asked, rather than filtered out of its answer afterwards.
  • Auditability: Prompt, generated SQL and user identity are written to an audit log. Natural-language querying is never an open door onto raw warehouse tables.
A single 1,800-line legacy script decomposed into sources, staging, intermediate and mart layers, each with its own tests and generated documentation.
Pattern 04

ETL processes and dbt modeling

  • Extraction into typed tables: APIs, PDFs and semi-structured JSON parsed into typed staging models with validation tests attached at the point of landing.
  • Legacy SQL to dbt: Large scripts decomposed into layered models with tests and docs — so a broken number names the layer that broke, not a file.
  • Streaming where it pays: Kafka or cloud event buses into the lakehouse with idempotent consumers and lag monitoring. We will tell you when a nightly batch is the right answer.
A pull request branches off main through a CI run and review; the same run regenerates the data dictionary, model documentation, lineage and the compliance audit entry.
Pattern 05

Documentation and metadata generation

  • Generated data dictionaries: Documentation regenerated by the same pipeline run that validated the change, so it cannot drift from the models it describes.
  • Compliance audit trail: The record of who changed which transformation, and when, is a build artefact rather than something somebody remembers to write down.
  • Feedback on data products: Usage and feedback on internal dashboards analysed to find broken metrics and the reports nobody opens.
Legacy Oracle columns mapped to Snowflake columns, with two rows crossing because value sampling — not name similarity — decided the match, and one legacy column having no target at all.
Pattern 06

Schema matching and entity resolution

  • Cross-system entity resolution: Customer records reconciled across Salesforce, SAP, NetSuite and the warehouse, with a confidence score attached to every proposed match.
  • Semantic schema alignment: Legacy Teradata and Oracle schemas mapped to target schemas on meaning and sampled values, because column names lie more often than data does.
  • A threshold, and a person: Above the confidence threshold we map automatically; below it a data owner decides. Silence is never treated as agreement.
A masking policy bound to a PII tag attaches to a column, so an analyst querying the table receives a masked value while an entitled approver receives the raw value and the read is logged.
Pattern 07

Data governance and automated PII masking

  • PII and PHI discovery: Pattern detectors plus classifiers to find sensitive fields in structured columns and in free text, where most of it actually hides.
  • Masking bound to the column: Snowflake and Databricks masking policies attached to the column, so the control travels with the data instead of living in a report filter someone can remove.
  • Entitled access is still logged: Seeing the raw value is a permission, not a bypass — the read is recorded either way.
Customer, account, order, product, supplier and shipment entities connected by named relationships such as "places", "contains" and "sourced from".
Pattern 08

Knowledge graphs and enterprise ontologies

  • Graph assembly: Graph databases — Neo4j or Amazon Neptune — connecting customers, products and supply-chain nodes that currently only meet inside a join.
  • The relationship is data too: “Sourced from” becomes a queryable fact with its own properties, rather than something implied by a foreign key and a tribal convention.
  • Ontology alignment: Domain ontologies reconciled across merger and acquisition datasets, which is usually where two companies discover they define “customer” differently.
A proposed column drop traced forward through staging and intermediate models to three downstream consumers — an executive board report, an operations report and a margin API — each with a named owner.
Pattern 09

Automated lineage and impact analysis

  • End-to-end lineage: Traced from source webhooks through to the executive dashboard, parsed from pipeline code so it cannot quietly go stale the way a hand-drawn diagram does.
  • Impact before migration: Simulate a schema change and see every model, metric and report that breaks — and who owns each one — before the migration is approved.
  • Named owners, not a broadcast: The output of an impact analysis is a list of people to talk to, which is the part that actually prevents the outage.
Invoice, purchase order and goods receipt converge on a three-way match; matched items post through an idempotent ERP adapter while exceptions are packaged with evidence for an approver.
Pattern 10

Autonomous AP exception resolution

  • AP exception agents: Invoices matched to purchase orders and receipts; anything outside tolerance is packaged with its evidence and a recommended action for a human approver.
  • No unapproved writes: The ERP adapter is idempotent and gated. The agent prepares the entry; a person approves it; a retry cannot double-pay.
  • Close readiness: Continuous reconciliation so that close week is confirmation rather than discovery. We baseline your cycle before agreeing a target — see the DataOps and finance masterclass for why we will not quote you a number first.
A Git repository holding desired state feeds a reconciler that applies it to dev, staging and production clusters, corrects drift on a loop, and stops at a human gate for production IAM and network changes.
Pattern 11

Kubernetes and GitOps infrastructure automation

  • Platform reliability: HPA and KEDA scaling, GitOps promotion with Argo CD or Flux, and policy as code. Agents help with runbooks and alert noise; they do not replace SRE fundamentals.
  • Drift detection: Detected through plan pipelines and corrected automatically only where that was explicitly approved — production networks and IAM stop at a person.
  • Rollback is a revert: Because desired state lives in version control, undoing a bad change is a git operation rather than an incident-time improvisation.
A discharge summary parsed into validated Patient, Observation and Condition resources behind a role-bound access gate that logs every read, with anything failing validation held rather than written.
Pattern 12

Healthcare and EHR document parsing

  • Clinical document structuring: Notes and lab results extracted into FHIR-oriented resources with validation — always under a business associate agreement and least-privilege access.
  • Invalid records are held, not guessed: Anything that fails validation is held for review rather than written with a plausible-looking value filled in.
  • Privacy-aware retrieval: Retrieval runs only over approved corpora, every read is logged, and customer PHI is not used for training unless a contract specifically scopes it.

These are patterns, not products. What any of them costs and how long it takes depends on your data, your access approvals and your change control — so we audit first.

Start with an audit