1. A metric you do not use is not a metric
A scorecard exists to change a decision. Ship, hold, roll back, or stop. A dashboard nobody opens is decoration. Pick the few numbers that would make you act, and look at them on a cadence a human will keep. Weekly is enough for a pilot. Daily is for a page that is already in the path of real work.
If a number does not move and the program claims success, the program is not measuring the thing it claims. The finance masterclass makes that point about a close. It applies here too.
2. Seven questions, not seventy charts
Quality. Is the output acceptable? Correctness, whether it is grounded, whether the citation supports the sentence, whether it abstained, and how often a person had to correct it. Retrieval. Did it find the right evidence? Recall, whether an exact identifier was found, and whether the reranker helped. The path. Did the workflow reach the goal? Task success, the right tool, retries, how many steps, and whether it recovered.
Security. Are the controls working on the tests you know, and on what production is doing? Injection regressions, policy violations, unauthorized actions, provenance failures. Speed and capacity. Time to first token, end-to-end latency, throughput, utilization, against the spare capacity you promised. The bill. Cache and batch hit rates, the full monthly cost, and cost per accepted outcome. Adoption. Active users, completed tasks, how often people override or bypass, how hard the review is, and whether cycle time actually moved.
3. How to read a bad week
Quality down and retrieval down usually means the evidence, not the adjective in the prompt. Quality down and retrieval steady usually means the instruction, the model, or the judge. Cost per accepted outcome up, with the same quality, means retries, a cache you lost, or a route that got more expensive. Adoption down while quality holds means the workflow does not fit the job. People route around a tool that adds a step and removes none.
Security failures are not an average you smooth. One unauthorized action is a stop, a trace, and a case in the eval set. Do not wait for the weekly meeting to notice it. The weekly meeting is where you confirm the case was added.
4. Keep the series in one place
- A model is not a system
- Local, sovereign, and hybrid
- The loop inside the harness
- Context, and when it goes stale
- A human at the irreversible step
- Evals and release gates
- What a run actually costs
- Six things a company can build
- A reference picture and a first roadmap
- A weekly scorecard
The Claude Code series, for the person at the keyboard, starts at an agent is a model, tools, and a loop.
5. What to put on the page this Friday
Three numbers are enough for the first Friday: how many outcomes were accepted, what each accepted outcome cost, and how often a person bypassed the tool. If you cannot collect them, the pilot is not observable yet. Fix that before you add a fourth chart.