ENTERPRISE AI

Context, and when it goes stale

The window is a budget. Longer input can get less reliable. Retrieve the evidence for this step, and test that choice on your own workload.

Figure 1: The context window is a budget. Long context goes stale. Retrieve the evidence for this step.

1. The window is a budget, not a library

The context window is how much text the model can consider at once. Some of it is spent before the user types, on instructions and tool definitions. The rest is the task, the evidence, and the history. Filling it with a whole archive feels thorough. It crowds out the paragraph that mattered.

Context rot is the name for a measured effect: as input length grows, reliability does not degrade in a neat straight line, and it does degrade. A 2025 technical report from Chroma is one public write-up. Do not treat it as a token number that is safe for every model. There is no universal safe length to publish here.

2. Summaries and full history disagree, on purpose

One published tutoring study found that keeping a stable full history beat the summarization methods they tested, once prompt caching was working. Other research found gains from compressing context for long-running agents. Those are different results on different workloads. They are not two votes for one rule.

Caching changes the economics and the behavior. Providers differ on what can be cached, how long a prefix must be, when it expires, and whether a new session keeps it. Read the current docs for the provider you pay. Then test on your tasks. A strategy copied from a talk is a hypothesis.

Layman rule: Pick a context strategy by a test on your workload. Do not pick it by a diagram you liked.

3. Retrieve what this step needs

Retrieval puts evidence into the window instead of hoping the model remembers the corpus. When the identifier must be exact, a case number or a SKU, combine semantic search with a lexical method. Meaning search finds neighbors. It misses the exact string. A reranker can reorder the candidates. It is not a substitute for access control. The agent must not retrieve a document the user is not allowed to see, then summarize the secret into the answer.

Every retrieved passage needs provenance: where it came from, and how old it is. A summary must not become the only record. Keep the source. Stale context is a specific failure. The policy changed, the index did not, and the model cited the old page with confidence.

4. Compaction drops detail

Compacting a long conversation saves the window and loses the exception you stated once. If the finish line, the non-goal, or the source of a number matters, write it into structured state or a file. Do not leave it only in a thread you are about to summarize.

Abstain when the evidence is not there. An answer that admits it cannot support a claim is more useful than a fluent paragraph with a citation that does not say what the sentence claims. Check citations against the source, not against the fact that a citation mark is present.

5. What to try next

Take ten real questions. Run them with the full packet you use today, and with a retrieved packet limited to the evidence for that question. Keep the strategy that fails less often on the questions you care about. Write down the date of the index.

Some steps still should not complete alone: a human at the irreversible step.