---
theme: seriph
title: "Lesson 22 — How Do You Test an Agent Instead of Its Final Answer?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 6 · Agent Evaluation
# How Do You Test an Agent Instead of Its Final Answer?
Environments, state, datasets, and executable verification
Lesson 22 of 42 · 18 minutes · Automated Evaluation Environment; Evaluation Task Datasets; Simulation Environments
---
Improve · Chapter 6 · Agent Evaluation
# Problems this chapter will solve
Lesson 22
How Do You Test an Agent Instead of Its Final Answer?
Lesson 23
How Do You Judge Quality Without Hiding Failure?
Lesson 24
Which Agent Should You Ship?
Lesson 25
Did the Agent Improve—or Did the Numbers Move?
---
# Why this problem matters
Initial state
Every run begins from the same controllable world.
Interaction
The Agent receives realistic tools, errors, and user disclosures.
Verification
Success is read from external state—not self-reported.
---
# Three ideas to keep in view
Dataset
Initial state + goal + boundary cases + success criteria
Environment
State transitions, reset, tools, and termination
Protocol
Tool-only tasks or progressive user simulation
---
# The book's visual model
Tool-calling and human-computer interaction evaluation environments
---
# Answer benchmark vs. Agent evaluation
Answer benchmark
- One prompt
- One response
- String or judge score
Agent evaluation
- Mutable state
- Multi-turn trajectory
- Executable outcome checks
An Agent can say the right thing while changing the world incorrectly.
---
# Reset, act, and inspect
~~~python
state = environment.reset(case.seed)
trajectory = agent.run(case.goal, environment.tools)
outcome = environment.snapshot()
passed = verifier.check(outcome, trajectory)
store(case, trajectory, outcome, passed)
~~~
---
# Test the claim
Evaluation control2 min
Score a reporting Agent from external state
Observe: Tool-call correctness, arithmetic, evidence citations, and unsupported-claim vetoes
Demo budget: 2 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ cd chapter6/public-health-reporting-eval && python demo.py
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
A resettable environment turns a trajectory into a repeatable experiment.
Finding 2
Progressive disclosure tests whether an Agent knows what to ask.
Finding 3
Objective state checks are stronger than judging the final prose alone.
---
layout: center
---
Where the claim stops
# Boundary condition
A simulator is useful only within its fidelity envelope; its omissions and biases become the Agent's test world.
---
layout: center
---
Engineering takeaway
# Design rule
Define the initial state, allowed transitions, and external success check before writing evaluation prompts.
---
# Continue the experiment · 1/2
---
# Continue the experiment · 2/2
---
layout: center
class: text-center
---
Pause and apply
# Your turn
Which production state would reveal success even if the Agent's final message were hidden?
---
layout: center
class: text-center
---
Next · Lesson 23
Turn observable outcomes into metrics that are consistent, diagnostic, and hard to game.
→