A8gent
GlossaryAgent concepts

What is eval?

Also called: evaluation, AI evals, model evaluation

An eval is a repeatable test that runs your AI system against a fixed set of example inputs and scores the outputs, so you can tell whether a change made things better or worse.

Written by A8gent
Contents
  1. At a glance
  2. What an eval is
  3. How to build one that is actually useful
  4. Cost, cadence and what to track
  5. What you should know
  6. FAQ
  7. Related terms

At a glance

Category
Core agent concept
What it is
A fixed set of inputs plus expectations plus a scoring method
Good starting size
20 real cases pulled from production logs, including the awkward ones
Scoring methods
Exact match, assertions, model-graded rubric (verify against humans first)
Run it on
Every prompt change, every model swap, plus a schedule against live traffic
Track alongside quality
Cost per case and p95 latency, so improvements do not hide a bill

What an eval is

An eval is the AI equivalent of a test suite: a fixed set of inputs, an expectation for each, and a scoring method. You run it before and after any change - a new prompt, a cheaper model, an extra tool - and compare the scores. Without one you are tuning by anecdote, which is how teams end up shipping a "better" prompt that quietly broke the case they never tested.

The reason evals matter more here than in ordinary software is that the system is non-deterministic and the failure is often subtle. A regression does not throw an exception; it produces a slightly worse summary, or misclassifies one lead type in twenty. You only see it in aggregate, which is exactly what an eval measures.

How to build one that is actually useful

Start with twenty real examples, not synthetic ones. Pull them from the logs of the thing you are automating: real enquiries, real invoices, real voicemails. Include the awkward ones - the empty message, the person who asked two questions at once, the customer who replied in another language. A set of twenty real cases catches more regressions than two hundred clean synthetic ones.

Then choose a scoring method per case. Exact match works for classification and extraction, and it should be your default because it is cheap and unambiguous. For open-ended output, use assertions rather than a single grade: does the reply mention the price, does it avoid promising a delivery date, is it under 300 characters. Reserve model-graded scoring - an LLM judging outputs against a rubric - for the genuinely subjective cases, and check it against human judgement on a sample before you trust it.

Cost, cadence and what to track

Running an eval costs the same as running the system, times the number of cases, times the number of variants you compare. Twenty cases against three prompt variants on a cheap model is pennies. That is why the size of the set is rarely the constraint - discipline is.

Run it on every prompt change, every model swap, and on a schedule against production traffic, because the world drifts even when your code does not. Track a small number of numbers over time: accuracy on the classification cases, assertion pass rate on the open ones, average token cost per case, and p95 latency. The cost and latency columns are what stop a quality improvement quietly tripling your bill.

What you should know

Your best eval data is free and you are probably throwing it away. Every time a person edits or rejects a draft in a human-in-the-loop step, that is a labelled example of what the system got wrong and what right looks like. Capture the before and after, and the eval set builds itself.

Finally, do not confuse evals with public benchmarks. A model topping a leaderboard tells you almost nothing about whether it can classify your enquiries correctly; your twenty real cases tell you exactly that. Evals are also the practical foundation of observability: the traces you log in production are what you replay when you want to know whether last month's build would still pass today.

Eval FAQ

What is an eval in AI?

A repeatable test: fixed example inputs, an expected result or set of assertions for each, and a score. You run it before and after a change so you can prove the change helped rather than assuming it did from a couple of hand-tried examples.

How many examples do I need in an eval set?

Twenty real ones beats two hundred invented ones. Take them from the logs of the process you are automating and deliberately include edge cases - blank fields, two questions in one message, an unusual format. Grow the set every time production produces a failure you did not have covered.

Can I use an LLM to grade my eval?

Yes, for subjective outputs where exact matching is impossible, but validate the judge first: grade thirty cases by hand and check the model agrees. Judges drift with model versions, so pin the judge model and re-verify when you upgrade it. For classification and extraction, exact match is cheaper and more trustworthy.

How often should I run evals?

On every prompt or model change before it ships, and on a schedule - weekly is common - against a sample of live traffic. Scheduled runs catch drift that your code did not cause: providers update models, your customers change how they write, and your data changes shape.

Do public benchmarks tell me which model to use?

Only roughly. Leaderboards measure general capability on tasks that are nothing like yours. A small model can beat a frontier one on your specific classification job at a tenth of the cost, and the only way to find out is to run your own eval set across both.

Where do I get labelled data for evals without extra work?

From your human approval step. Every edit or rejection is a labelled pair of what the system produced and what a person actually wanted. Store both, and your eval set grows as a by-product of running the automation rather than as a separate project.

Primary source

OpenAI - Evaluations guide

Checked August 2026. Anything priced or rate-limited moves, so verify against the source before you budget.

Want this built rather than explained?

Knowing what eval means is one thing; having it running in your business is another. We scope one workflow at a time, quote a fixed price in writing after a free consult, build it, and hand it over with documentation you own.