---
title: Agent Evals
description: Agent evals are an automated test suite for agent behaviour — a golden dataset of cases with expected outcomes and a pass bar the agent must clear before deployment.
url: https://www.factualminds.com/glossary/agent-evals/
publishDate: 2026-08-30
updateDate: 2026-08-30
---

# Agent Evals

> Agent evals are an automated test suite for agent behaviour — a golden dataset of cases with expected outcomes and a pass bar the agent must clear before deployment.

## Definition

**Agent evals** are an automated test suite for agent behaviour: a curated set of representative cases (a **golden dataset**), the expected outcome for each, and a scoring method that produces a number the agent must clear before it ships.

An agent without evals is a system whose quality you are **asserting** rather than **measuring**. That distinction is what separates agents still running after ninety days from agents that were quietly switched off.

## What goes in a golden dataset

- **The boring majority.** The cases that make up most of the volume, where a wrong answer is expensive precisely because it is frequent.
- **The known-hard cases.** Split shipments, partial refunds, contract pricing, anything where a competent human hesitates.
- **The adversarial cases.** Prompt injection attempts, out-of-scope requests, and inputs designed to make the agent exceed its remit.
- **The cases that already broke.** Every production incident should end with a new eval case. This is the single highest-return habit.

Fifty well-chosen cases beat five hundred generated ones.

## The pass bar

The number matters less than the fact that it is agreed **before** the results are in, and that failing it blocks deployment. A pass bar renegotiated after a bad run is not a pass bar.

Set it per agent and per action class. A support agent answering WISMO questions and a margin agent proposing price changes should not share a threshold, because the cost of being wrong differs by orders of magnitude.

## Where they run

On AWS, AgentCore Evaluations runs the suite as part of the deployment path, alongside traces from AgentCore Observability. The important property is that it is **in the pipeline**, not a notebook someone runs when they remember.

## Limits worth stating

- **Evals measure what you thought to test.** They are a floor, not a guarantee. Novel failure modes arrive in production first, which is why observability and the incident-to-eval habit matter.
- **Scoring open-ended output is genuinely hard.** Exact-match works for structured decisions; for free text you are choosing between a rubric, a model-as-judge (with its own bias), and human review of a sample. Each has failure modes; none is free.
- **A passing suite does not mean the agent is safe.** Safety comes from the [tool catalog](/glossary/tool-catalog/) and authorization. Evals tell you whether it is *good*.

## Related terms

[AI agent](/glossary/ai-agent/) · [Tool catalog](/glossary/tool-catalog/) · [Human in the loop](/glossary/human-in-the-loop/) · [Amazon Bedrock AgentCore](/glossary/bedrock-agentcore/)

## Related AWS Services

- amazon-bedrock-agentcore
- ecommerce-ai-agents
- generative-ai-on-aws

## Related Posts

- bedrock-llm-observability-evals-harness-2026
- production-ai-agents-aws-agentcore-harness-strands-2026
- amazon-bedrock-agentcore-production
- ecommerce-ai-implementation-challenges-2026

---

*Source: https://www.factualminds.com/glossary/agent-evals/*