When the rehearsal becomes the exam
Anthropic's January 9, 2026 evaluation article distinguishes a task, repeated trials, grading, and the final state left by an agent. Google's machine-learning course explains a related problem: repeatedly using the same validation or test examples to make improvements can weaken their value as evidence about new data. These are background references, not this week's launches. Silver Owl's inference is that an AI workflow needs some examples reserved from everyday tuning. Otherwise, a rising score may reflect a better rehearsal without showing how the system handles unfamiliar work.
Source: Demystifying evals for AI agents · 2026-01-09
Give each example a clear role
For a modest pilot, divide the collection by purpose before experimentation begins. Use development examples to inspect mistakes and improve the workflow. Use a separate comparison set to track whether a change helps or harms known cases. Reserve a final group for an evaluation after the candidate configuration is fixed. These are recommended operating roles, not a universal numerical split. The right collection depends on the work and its consequences. Record who can inspect each group and when a previously reserved example becomes part of routine development.
Separate related cases together
Google's course warns that duplicate examples across development and test collections can distort an evaluation. For business documents, our suggested extension is to examine close relatives too. If several examples come from the same customer thread or nearly identical form, consider keeping the family together when splitting the collection. This is an editorial method for reducing accidental rehearsal, not a claim about a particular model's training data. Document the grouping rule so another reviewer can check it. A random split deserves inspection when the original material contains repetition.
Freeze the candidate, then run the comparison
Write down the model, instructions, connected tools, source material, and important settings before a comparison run. Anthropic's distinction between trials and outcomes is useful here: an agent's closing message is not the same evidence as the state it leaves behind. Our recommendation is to inspect the actual artifact or permitted system result, and repeat representative cases where variation matters. Record each attempt, including incomplete ones. If the team changes the configuration midway through the run, label the result as a different candidate rather than blending the scores into one cleaner-looking number.
Source: Demystifying evals for AI agents · 2026-01-09
Keep failures visible after they become familiar
Suppose a fictional document assistant repeatedly confuses an amended delivery date with the original date. Once the team studies that example and repairs the workflow, retain it as a regression check. It is still valuable, but it is no longer untouched evidence. Add a different representative case to the reserved collection when appropriate, with its expected outcome reviewed independently. Maintain a small change history showing why examples moved between groups. This avoids the temptation to discard difficult cases merely because they make the headline score less attractive.
Report the limits alongside the result
A useful evaluation note states how many cases were attempted, which work they represent, which cases were excluded, and whether developers had seen them. Describe serious failures separately from the overall score. Keep privacy and retention rules in force when collecting examples; realistic does not mean unrestricted. Silver Owl's recommendation is to make the evidence honest enough for a decision: what improved, what remained weak, and what the collection cannot tell you. A protected holdout is one part of that discipline, alongside representative cases, competent review, and observation after a supervised release.