Lesson 23 — Evaluating an agent with test cases

How do you know it works?

An eval (evaluation) is a list of test cases: an input and what a good answer must contain. You run the agent on every case and count the passes. This gives you a score like 4/5.

Real model answers vary in wording, so evals usually check that the answer contains the key fact ("Paris" in answer) rather than matching it word for word.

Run the evals every time you change a prompt, a tool or the model. If the score drops, you broke something.

Watch out: when a case fails, fix the agent — not the test. The Tokyo case below fails on purpose: it shows you what to improve next.

Why it matters: without evals, "it seems to work" is a guess. With evals, it is a number you can track.