AI AGENT ARCHITECTURE

Architecting Production AI Agents at Enterprise Scale

A plain-language blueprint for building AI helpers that do real work inside your company — not demos that break on Monday morning.

Figure 1: A production agent runtime reaching the ERP, CRM and warehouse systems of record through safe actions, writing checkpoints to three kinds of memory, and handing uncertain cases to a human approve / reject gate.

1. What “production agent” really means

When people say “AI agent,” they often mean a chatbot that answers questions. That is useful — but it is not what enterprises need for day-to-day operations.

A production agent is software that can:

  • See what is happening (a new invoice, a late shipment, a support ticket),
  • Decide the next safe step using rules plus AI judgment, and
  • Do something useful in your systems (update a record, request approval, open a ticket) — and leave an audit trail of what was proposed and what a person approved.

In short: it is less like a clever search box and more like a careful junior employee who does not get tired, and who is required to ask a human when confidence is low or the action is high risk.

In one sentence: Production agents automate multi-step work inside your existing tools, with safety rails, memory of progress, and clear handoffs to people.

2. Why chatbots fail at real work

Early pilots stall for boring reasons — not because the model is “not smart enough.” We do not have a published count of how often. Orchestration libraries such as LangGraph or LlamaIndex workflows run the steps. They do not replace the engineering around which tools are allowed, how a side effect stays safe to retry, how you tell a good change from a bad one, or where the audit trail lives. LangGraph can save graph state when you attach a checkpointer. That saved state is not an evaluation set, and it is not the audit log.

Typical failure modes:

  • No reliable memory. The system forgets what it already did if the browser refreshes or a vendor API times out.
  • No safe actions. The bot can talk about fixing an invoice but cannot update the finance system (or worse, updates it without checks).
  • No ownership of exceptions. When something is unclear, the issue disappears into a dashboard nobody watches until month-end.
  • Demo data only. Real invoices are messy: missing fields, PDFs, mismatched purchase orders, foreign currencies.

Building agents for the enterprise is therefore a software engineering problem first — reliability, security, observability — and a model-prompting problem second.

3. Three kinds of memory (simple view)

Think of each agent’s memory the way a careful office worker organizes a desk — not as one big notepad, and not as one super-bot, but as three drawers:

  1. Scratchpad (short-term). What the AI is thinking about right now — like sticky notes on a desk. This is the model’s temporary context. It should not be the system of record.
  2. Checkpoint (in-progress work). “I already matched the invoice header; I still need the line items.” This lives in a fast database, for example Redis or PostgreSQL, so work can resume after a failure.
  3. Company memory (history & analytics). Approved outcomes, audit logs, and trends stored in a data platform such as Snowflake or Databricks. Definitions can live in code with a tool such as dbt, where a person can review them. The tool does not make a number trustworthy by itself. Someone still owns the definition.
Figure 2: Three kinds of memory — scratchpad (model context window), checkpoint (Redis / PostgreSQL, resume after failure) and company memory (Snowflake / Databricks + dbt, approved outcomes for audit) — three drawers, not one big notepad.

Why separate them? Because if you store everything only in the AI’s chat window, you lose work when sessions end, you cannot audit decisions cleanly, and you pay again every time the model re-reads the same history.

Memory type Everyday analogy Typical tech What it is for
Scratchpad Sticky notes on a desk Model context window Assemble the next decision
Checkpoint Folder of “work in progress” Redis / PostgreSQL Resume from the last finished step
Company memory Filing cabinet + reporting Snowflake / Databricks + dbt Audit, analytics, and history you can query

4. Teams of agents, not one super-bot

One giant agent that tries to parse PDFs, write SQL, call APIs, and email vendors at once becomes hard to test and harder to trust.

When Cloudadorn embeds engineers, we design small specialist agents that work as a team:

  • Worker agents do one job each (extract fields, check policy, fetch vendor history). Some of those jobs can run at the same time.
  • A coordinator passes clean, structured results from one step to the next.
  • A judge agent (or hard-coded rules) blocks unsafe writes — for example, “do not post this payment if confidence is below 95% or the amount exceeds policy.” Blocking a bad write is not permission to post a good one. A person still approves before money moves.

This mirrors how human teams already work: specialists, handoffs, and a final approver for high-risk steps.

5. Exceptions to people — not dashboards

The failure mode we design against is finding out too late, from a red chart on a dashboard, after customers are already waiting.

Production agents should reverse that:

  • When the agent is unsure, it packages the problem (what it saw, what it recommends, what is blocked).
  • It sends that package to a person in Slack, Teams, or email, with a direct approve or reject. The point is a decision, not another dashboard to hunt through.
  • Every decision is logged for audit (who approved, when, and why).

People stay in control of judgment. Software removes the copy-paste and the hunting for context.

6. A practical build checklist

Whether you embed engineers from Cloudadorn or use your internal team, follow this order:

  1. Pick one painful workflow with clear dollar or time impact (e.g., AP exceptions, claims triage, shipment delays).
  2. Map systems of record — where truth lives today (ERP, CRM, warehouse, ticket tool).
  3. Define safe actions — what the agent may do alone vs. what needs a person.
  4. Add checkpoints — save progress after each major step.
  5. Measure — time saved, exception rate, false approvals, cost per run.
  6. Expand — only after the first workflow is boringly reliable.
Business takeaway: Start narrow, make it trustworthy, then scale. We refuse wide pilots that skip safety and memory. We do not have a published count of how often those stall. The pattern is familiar enough that we will not start there.

7. Code snapshot (for engineers)

The idea below is simple: save structured progress after each step so a crash does not throw away work that already finished. A real runner reads checkpoint_step and continues from there. The snippet shows only the save and the load. amount is a decimal, not a binary float, because many cent amounts are not exact in binary. The Redis key holds the latest draft and the next save overwrites it. It is not the audit log. Who approved what belongs in company memory, where a later save does not erase it.

agent_checkpoint_engine.py Python 3.11
from decimal import Decimal

import redis
from pydantic import BaseModel

class InvoiceState(BaseModel):
    invoice_id: str
    amount: Decimal
    vendor_id: str
    status: str
    checkpoint_step: int

class ProductionAgentRunner:
    def __init__(self, redis_url: str):
        self.r = redis.from_url(redis_url)

    def save_checkpoint(self, session_id: str, state: InvoiceState):
        # Latest draft only. The next save overwrites this key.
        self.r.set(f"agent:checkpoint:{session_id}", state.model_dump_json())

    def resume_execution(self, session_id: str) -> InvoiceState:
        raw = self.r.get(f"agent:checkpoint:{session_id}")
        if not raw:
            raise ValueError(f"No checkpoint for session {session_id}")
        return InvoiceState.model_validate_json(raw)
            

Leaders do not need to read the code — the point is: progress is saved like a document draft, not only inside a chat window.