Use cases

AI & data use cases

Twelve patterns we implement in production. Each one is drawn with the control that makes it safe to run — the gate, the threshold, the person who approves — because that part is the actual engineering, and it is what a demo leaves out.

Records from files, APIs and change-data-capture pass through a validation gate that routes them to load unchanged, normalize then load, or quarantine for human review.
Pattern 01

Data cleaning and preprocessing

  • Anomaly and gap detection: Rules plus ML and LLM classifiers flag null spikes, outliers and schema drift before a load lands in Snowflake or Databricks — not after a dashboard goes wrong.
  • Text normalization: Abbreviations, locales and free-text synonyms standardized against an agreed dictionary, with human review for anything the rules are not confident about.
  • Code mapping: Supplier and vendor codes aligned to internal MDM keys — deterministic maps first, fuzzy and semantic matching only for what is left over.
A classifier reads column names, sample values and existing documentation, applies confident tags automatically and sends the rest to a steward queue.
Pattern 02

Data cataloging and automated tagging

  • Semantic tagging: Column content — not just column names — classified with embeddings and rules, so notes_free and cust_id are not treated the same way.
  • Metadata extraction: Structural and operational metadata pulled from PDFs, object storage and legacy database schemas into one catalog.
  • Tags that do something: A tag is not documentation: applied tags are what the masking policies in the governance pattern below actually bind to.
A question passes through a semantic layer of certified metrics and allow-listed models before SQL is generated; requests for raw PII or cross-tenant joins are refused, and every prompt is logged.
Pattern 03

Natural language querying and semantic access

  • Natural language to SQL: Questions routed through a governed semantic layer — dbt metrics and semantic models — so generated SQL can only touch approved tables and approved metric definitions.
  • Refusal before generation: Cross-tenant joins and unmasked PII columns are blocked by policy before the model is asked, rather than filtered out of its answer afterwards.
  • Auditability: Prompt, generated SQL and user identity are written to an audit log. Natural-language querying is never an open door onto raw warehouse tables.
A single 1,800-line legacy script decomposed into sources, staging, intermediate and mart layers, each with its own tests and generated documentation.
Pattern 04

ETL processes and dbt modeling

  • Extraction into typed tables: APIs, PDFs and semi-structured JSON parsed into typed staging models with validation tests attached at the point of landing.
  • Legacy SQL to dbt: Large scripts decomposed into layered models with tests and docs — so a broken number names the layer that broke, not a file.
  • Streaming where it pays: Kafka or cloud event buses into the lakehouse with idempotent consumers and lag monitoring. We will tell you when a nightly batch is the right answer.
A pull request branches off main through a CI run and review; the same run regenerates the data dictionary, model documentation, lineage and the compliance audit entry.
Pattern 05

Documentation and metadata generation

  • Generated data dictionaries: Documentation regenerated by the same pipeline run that validated the change, so it cannot drift from the models it describes.
  • Compliance audit trail: The record of who changed which transformation, and when, is a build artifact rather than something somebody remembers to write down.
  • Feedback on data products: Usage and feedback on internal dashboards analyzed to find broken metrics and the reports nobody opens.
Legacy Oracle columns mapped to Snowflake columns. CUST_NO and CUST_NM map straight across. DT_CREATE maps to created_at but the type change needs a person. ADDR_L1 maps to address_line_1 because sampled values confirm it. LEGACY_FLG has no target.
Pattern 06

Schema matching and entity resolution

  • Cross-system entity resolution: Customer records reconciled across Salesforce, SAP, NetSuite and the warehouse, with a confidence score attached to every proposed match.
  • Semantic schema alignment: Legacy Teradata and Oracle schemas mapped to target schemas on meaning and sampled values, because column names lie more often than data does.
  • A threshold, and a person: Above the confidence threshold we map automatically; below it a data owner decides. Silence is never treated as agreement.
A masking policy bound to a PII tag attaches to a column, so an analyst querying the table receives a masked value while an entitled approver receives the raw value and the read is logged.
Pattern 07

Data governance and automated PII masking

  • PII and PHI discovery: Pattern detectors plus classifiers to find sensitive fields in structured columns and in free text, where most of it actually hides.
  • Masking bound to the column: Snowflake and Databricks masking policies attached to the column, so the control travels with the data instead of living in a report filter someone can remove.
  • Entitled access is still logged: Seeing the raw value is a permission, not a bypass — the read is recorded either way.
A customer holds an account and places an order. The order contains a product and ships as a shipment. The product is sourced from a supplier. The relationship is stored, not implied by a join.
Pattern 08

Knowledge graphs and enterprise ontologies

  • Graph assembly: Graph databases — Neo4j or Amazon Neptune — connecting customers, products and supply-chain nodes that currently only meet inside a join.
  • The relationship is data too: “Sourced from” becomes a queryable fact with its own properties, rather than something implied by a foreign key and a tribal convention.
  • Ontology alignment: Domain ontologies reconciled across merger and acquisition datasets, which is usually where two companies discover they define “customer” differently.
A proposed column drop traced forward through staging and intermediate models to three downstream consumers — an executive board report, an operations report and a margin API — each with a named owner.
Pattern 09

Automated lineage and impact analysis

  • End-to-end lineage: Model-to-model edges come from the pipeline code and regenerate on each build, so they do not drift the way a hand-drawn diagram does. A dashboard or API is on that graph only when an exposure names it.
  • Impact before migration: Simulate a schema change and see every model, metric and report that breaks — and who owns each one — before the migration is approved.
  • Named owners, not a broadcast: The output of an impact analysis is a list of people to talk to, which is the part that actually prevents the outage.
Invoice, purchase order and goods receipt converge on a three-way match; matched items post through an idempotent ERP adapter while exceptions are packaged with evidence for an approver.
Pattern 10

Autonomous AP exception resolution

  • AP exception agents: Invoices matched to purchase orders and receipts; anything outside tolerance is packaged with its evidence and a recommended action for a human approver.
  • No unapproved writes: The ERP adapter is idempotent and gated. The agent prepares the entry; a person approves it; a retry cannot double-pay.
  • Close readiness: Continuous reconciliation so that close week is confirmation rather than discovery. We baseline your cycle before agreeing on a target — see the DataOps and finance masterclass for why we will not quote you a number first.
A Git repository holding desired state feeds a reconciler that applies it to dev, staging and production clusters, corrects drift on a loop, and stops at a human gate for production IAM and network changes.
Pattern 11

Kubernetes and GitOps infrastructure automation

  • Platform reliability: HPA and KEDA scaling, GitOps promotion with Argo CD or Flux, and policy as code. Agents help with runbooks and alert noise; they do not replace SRE fundamentals.
  • Drift detection: Detected through plan pipelines and corrected automatically only where that was explicitly approved — production networks and IAM stop at a person.
  • Rollback is a revert: Because desired state lives in version control, undoing a bad change is a git operation rather than an incident-time improvisation.
A discharge summary parsed into validated Patient, Observation and Condition resources behind a role-bound access gate that logs every read, with anything failing validation held rather than written.
Pattern 12

Healthcare and EHR document parsing

  • Clinical document structuring: Notes and lab results extracted into FHIR-oriented resources with validation — always under a business associate agreement and least-privilege access.
  • Invalid records are held, not guessed: Anything that fails validation is held for review rather than written with a plausible-looking value filled in.
  • Privacy-aware retrieval: Retrieval runs only over approved corpora, every read is logged, and customer PHI is not used for training unless a contract specifically scopes it.

These are patterns, not products. What any of them costs and how long it takes depends on your data, your access approvals and your change control — so we audit first.

Start with an audit