Lesson 14 — Comparing two AI answers with tests

Let the tests decide

Ask the same question twice — or ask ChatGPT and Claude — and you will often get two different answers. Which one is better? Not the longer one, not the one that sounds more sure. The one that passes more of your tests.

A test suite is a list of test cases you run against any function:

def count_ok(func):
    ok = 0
    for inp, expected in EXAMPLES:
        if func(inp) == expected:
            ok += 1
    return ok

Because functions are values in Python, you can pass answer_a or answer_b into the same run_suite and compare the scores. Print the failures too — they tell you why an answer lost, and they are exactly what to send back to the assistant.

Why it matters: the same suite works for every answer, every retry and every assistant. It turns "I think B is better" into a number.