How to Evaluate AI Output: A Simple Test Set That Works

Language models write fluent text whether or not the content is correct. That makes "it looked good when I tried it" a very weak form of evidence. If an AI feature matters to your product, you need a small, boring evaluation process.

Start with real examples

Collect twenty to fifty real inputs from the task you care about: actual support questions, actual documents, actual code changes. Include the awkward ones: empty fields, mixed languages, contradictory instructions, very long inputs. Real data finds problems invented examples never will.

For each, write down what a good answer must contain, and what it must never contain. You do not need a perfect reference answer, only a clear rule for grading.

Grade against rules, not vibes

Prefer checks that can be automated:

• Does the output parse as the required JSON or schema?
• Does it cite only sources that were actually provided?
• Does it include forbidden content (private data, made-up links, wrong currency)?
• Does generated code compile and pass the tests?

Keep human review for the judgment calls (tone, usefulness), using a short rubric so two reviewers would agree.

Be careful with model-as-judge

Using one model to grade another is useful for scale, but it has known biases: it tends to prefer longer answers and answers that look like its own style. If you use it, calibrate: have a person grade a sample, compare, and only trust the automated grader where it agrees with people.

Track results over time

Every time you change a prompt, a model or a retrieval setting, rerun the set and compare. A change that improves three cases and silently breaks five is common, and you will not notice without numbers.

Store results in a simple table: input id, version, pass/fail, notes. A spreadsheet is enough to start.

Watch for these traps

• Testing on the examples you tuned on. Keep a hold-out set you never look at while editing prompts.
• Averages hiding severe failures. One answer that leaks private data matters more than ten slightly better answers. Count severe failures separately.
• Non-determinism. Run important cases more than once; a pass that happens one time in three is a fail.
• Drift. Models and data change. Re-run the set on a schedule, not only when you edit something.

Related reading

Before launch, pair this with the pre-launch checklist; for retrieval-based apps see RAG vs long context.

The minimum viable version

A file of 30 cases, a script that runs them and prints pass/fail, and the habit of running it before every change. That alone puts you ahead of most teams shipping AI features.