ENTERPRISE AI

A weekly scorecard

Once a week, look at quality, retrieval, the path the agent took, security, speed, cost per accepted outcome, and whether people use it.

Figure 1: A weekly scorecard looks at quality, cost per accepted run, and whether people use the system or bypass it.

1. A metric you do not use is not a metric

A scorecard exists to change a decision. Ship, hold, roll back, or stop. A dashboard nobody opens is decoration. Pick the few numbers that would make you act, and look at them on a cadence a human will keep. Weekly is enough for a pilot. Daily is for a page that is already in the path of real work.

If a number does not move and the program claims success, the program is not measuring the thing it claims. The finance masterclass makes that point about a close. It applies here too.

2. Seven questions, not seventy charts

Quality. Is the output acceptable? Correctness, whether it is grounded, whether the citation supports the sentence, whether it abstained, and how often a person had to correct it. Retrieval. Did it find the right evidence? Recall, whether an exact identifier was found, and whether the reranker helped. The path. Did the workflow reach the goal? Task success, the right tool, retries, how many steps, and whether it recovered.

Security. Are the controls working on the tests you know, and on what production is doing? Injection regressions, policy violations, unauthorized actions, provenance failures. Speed and capacity. Time to first token, end-to-end latency, throughput, utilization, against the spare capacity you promised. The bill. Cache and batch hit rates, the full monthly cost, and cost per accepted outcome. Adoption. Active users, completed tasks, how often people override or bypass, how hard the review is, and whether cycle time actually moved.

Layman rule: If the scorecard cannot tell you to stop, it is a status report.

3. How to read a bad week

Quality down and retrieval down usually means the evidence, not the adjective in the prompt. Quality down and retrieval steady usually means the instruction, the model, or the judge. Cost per accepted outcome up, with the same quality, means retries, a cache you lost, or a route that got more expensive. Adoption down while quality holds means the workflow does not fit the job. People route around a tool that adds a step and removes none.

Security failures are not an average you smooth. One unauthorized action is a stop, a trace, and a case in the eval set. Do not wait for the weekly meeting to notice it. The weekly meeting is where you confirm the case was added.

4. Keep the series in one place

The Claude Code series, for the person at the keyboard, starts at an agent is a model, tools, and a loop.

5. What to put on the page this Friday

Three numbers are enough for the first Friday: how many outcomes were accepted, what each accepted outcome cost, and how often a person bypassed the tool. If you cannot collect them, the pilot is not observable yet. Fix that before you add a fourth chart.