How should an AI agent be evaluated before production use?

Evaluate an AI agent as an **end-to-end system**, not only as a language model.\n\nA production evaluation should measure:\n\n- whether the final task was completed correctly;\n- correct tool selection and arguments;\n- factual grounding and citation accuracy;\n- policy, permission, and approval compliance;\n- recovery from tool failures and incomplete information;\n- resistance to prompt injection and malicious content;\n- cost, latency, turns, and retries;\n- consistency across repeated runs;\n- human review and correction effort;\n- harmful side effects and actions the agent should have refused.\n\nUse realistic private tasks, hidden checks, fixed environments, recorded traces, multiple runs, and comparison with a simple baseline. Continue evaluating after deployment with sampled production traces and regression suites.

An agent can produce an impressive final answer while taking an unsafe path. It can also make several harmless mistakes and still reach the correct result. Output-only evaluation therefore misses important information.\n\nA layered evaluation should include:\n\n### 1. Component evaluations\n\nTest individual elements in isolation:\n\n- query classification;\n- retrieval recall and ranking;\n- structured-output validity;\n- tool selection;\n- argument generation;\n- guardrails;\n- permission checks;\n- source attribution;\n- memory retrieval;\n- routing and handoffs.\n\n### 2. End-to-end task evaluations\n\nGive the system realistic tasks and score the final outcome. The environment should include the same tools, permissions, data shape, and constraints expected in production. Hidden tests should detect incomplete or superficially correct results.\n\n### 3. Trace evaluations\n\nInspect the sequence of model calls, searches, tool calls, errors, approvals, retries, and state transitions. Trace analysis can detect unnecessary loops, benchmark gaming, unsafe shortcuts, suspicious tool use, or accidental success.\n\nNIST has highlighted agent evaluation cheating and recommends clearer benchmark-specific expectations about agent affordances and restrictions. A system may otherwise exploit unintended environment behaviour rather than demonstrate the intended capability.\n\n### 4. Safety and adversarial evaluations\n\nTest prompt injection, misleading tool output, poisoned memory, inaccessible resources, permission boundaries, malicious documents, ambiguous targets, and requests that should require approval or refusal.\n\n### 5. Operational evaluations\n\nMeasure:\n\n- p50 and p95 latency;\n- total input and output tokens;\n- tool and infrastructure cost;\n- number of turns and retries;\n- timeout and abandonment rate;\n- human review time;\n- successful-task cost;\n- production incident and rollback rate.\n\n### 6. Statistical design\n\nOne run is not enough for stochastic systems. Repeat tasks across seeds or independent attempts, report confidence intervals, and avoid claiming meaningful differences from tiny samples. Keep model versions, prompts, tools, and environments recorded.\n\nNIST's AI 800-2 work emphasises validity, transparency, reproducibility, and documented benchmark practice. The evaluation question must match the deployment claim: a coding benchmark does not establish safe customer-service behaviour, and a general knowledge benchmark does not establish reliable tool use.\n\n### 7. Baselines\n\nCompare the agent against:\n\n- a deterministic workflow;\n- a single model without tools;\n- a simpler agent;\n- the existing human process;\n- a lower-cost model;\n- a human-plus-tool baseline.\n\nAn agent should not be deployed merely because it completes some examples. It should demonstrate a measurable advantage under an explicitly defined acceptable-risk threshold.\n\n### 8. Continuous evaluation\n\nModel updates, prompt changes, new tools, source changes, and shifting user behaviour can cause regression. Run the evaluation suite before releases, canary changes, monitor live traces, and convert production failures into new held-out tests.

- [NIST — Towards best practices for automated benchmark evaluations](https://www.nist.gov/news-events/news/2026/01/towards-best-practices-automated-benchmark-evaluations) — summarises NIST AI 800-2 guidance on validity, transparency, and reproducibility for model and agent evaluations. - [NIST — Cheating on AI agent evaluations](https://www.nist.gov/caisi/cheating-ai-agent-evaluations) — documents unintended benchmark exploitation and recommends standardising affordances and restrictions. - [NIST — Analyzing transcripts from AI agent evaluations](https://www.nist.gov/blogs/caisi-research-blog/analyzing-transcripts-ai-agent-evaluations) — establishes trace and transcript analysis as an evaluation method. - [NIST — Building evaluation probes into agentic AI](https://www.nist.gov/programs-projects/building-evaluation-probes-agentic-ai) — identifies factual grounding, completeness, and rubric-based evaluation of agent outputs. - [OpenAI — Evals API](https://platform.openai.com/docs/api-reference/evals) — documents reusable evaluation definitions, datasets, runs, and graders across models and settings.