Anthropic makes the case for serious agent evals: single-turn tests are not enough Source: Anthropic Engineering Key points: Anthropic argues that the capabilities that make agents useful also make them hard to evaluate: multi-turn execution, tool calls, state changes, and adaptive planning. A useful eval is not just a final answer score. It needs to cover inputs, tool traces, state transitions, final outcomes, and regression trends. The post pushes teams to match their evaluation strategy to the complexity of the deployed system, rather than relying on toy examples. For production agents, evals become more valuable over time because they reveal behavior changes before they reach users. Peon take: This is the most important read today. Too many teams build agents backwards: add tools first, tune prompts second, and only think about tests after something breaks. Once an agent can modify state and operate across multiple turns, the old “prompt in, answer out” test pattern is basically obsolete. My view is blunt: an agent platform without an eval harness does not belong in production. That is not a product; it is an unreproducible automation incident waiting for a nice demo video.