In brief
- Evals assess a specific application in its intended context, not only the underlying model.
- Good test cases cover common tasks, difficult edge cases and relevant risks from real work.
- Quality has several dimensions, including correctness, completeness, grounding, safety, latency and cost.
- Evals are repeated before launch, after changes and throughout operation.
What does an eval actually test?
An eval combines a clearly defined task, suitable inputs, expected properties of the result and a reproducible scoring method. It does not ask whether a model is generally intelligent. It asks whether a specific solution performs well enough under defined conditions.
Where possible, the complete application should be tested: prompt, model, retrieved sources, permissions, tools, business rules and output format. A model may perform well on a public benchmark yet still be unsuitable for a particular company process.
- Task success: Does the solution produce the result the process needs?
- Domain quality: Are statements correct, complete and traceable?
- Safety: Are prohibited content, actions and data access prevented?
- Operations: Do latency, stability and cost remain within agreed limits?
What belongs in a useful test set?
The test set should reflect actual use. A collection of simple sample questions is insufficient if day-to-day work includes incomplete information, conflicting documents, rare exceptions or ambiguous language.
Start with anonymised or realistically simulated cases from the intended process. Add failures found during pilots and production as regression tests so that a resolved problem does not silently return after a change.
- Frequent standard cases that represent most usage
- Business-critical cases where an error would have a serious impact
- Edge cases such as missing details, conflicting sources or unusual wording
- Negative cases where the solution should refuse or ask for clarification
- Adversarial and misuse cases, including manipulated documents or prohibited instructions
How are outputs scored?
The scoring method depends on the criterion. A structured field can be compared with an exact expected value. A domain-specific summary may require a rubric and review by subject-matter experts. Another language model can help score larger volumes, but it must first be calibrated against trusted human judgements.
A single overall score often hides important differences. A small set of quality dimensions with explicit thresholds and separate treatment of critical failures is more informative.
- Rule-based checks for format, required fields, numbers or allowed values
- Comparison with reference answers or approved source material
- Domain review by people who understand the process
- Model-assisted grading using a precise rubric
- Manual review of every high-impact failure category
What does a completed eval scorecard look like?
The following scorecard is a simplified example for an internal knowledge assistant with 100 versioned test cases. Thresholds are set before the run. A critical exclusion criterion must not be offset by strong averages in other dimensions.
| Criterion | Threshold | Example result | Decision |
|---|---|---|---|
| Answer supported by an approved source | At least 95 of 100 cases | 97 of 100 | Pass |
| All required claims present | At least 90 of 100 cases | 92 of 100 | Pass |
| Correct refusal without evidence | At least 9 of 10 negative cases | 8 of 10 | Fail: improve retrieval and refusal logic |
| Critical false contract statement | 0 cases | 1 case | Stop criterion: no release |
| Response time for 95% of requests | No more than 8 seconds | 6.2 seconds | Pass |
| Variable cost per request | No more than CHF 0.08 | CHF 0.05 | Pass |
How do you build a reliable evaluation process?
Evals are most useful when they begin before model or vendor selection. The team translates expected value and possible failure consequences into testable criteria. Prompts, models and architecture options can then be compared under the same conditions.
- Step 1
Define the task
Describe the work being supported, the required result and who will use it.
- Step 2
Define success
Set measurable criteria, minimum thresholds and unacceptable failures.
- Step 3
Collect cases
Create a versioned test set covering standard, edge, risk and negative cases.
- Step 4
Run and analyse
Review failure types and affected user groups, not only aggregate scores.
- Step 5
Protect changes
Rerun the same tests before release and make regressions visible.
Why does quality assurance continue after launch?
Models, prompts, data sources and user behaviour change. A production solution should therefore combine fixed test sets with real operational signals such as expert corrections, refusals, abandoned tasks, latency, cost and reported incidents.
Production data does not replace a controlled eval. It helps uncover new cases and shows whether the test distribution still matches actual use. Sensitive content must be anonymised where appropriate and handled according to access and retention rules.
- Version model, prompt and data changes
- Turn newly discovered failures into repeatable regression tests
- Analyse results by case type rather than only as one average
- Assign clear ownership for review, release and incident response
Example: evaluating a knowledge assistant
A service team builds a test set from anonymised questions about contracts, deadlines and internal procedures. Each case records the approved evidence together with required and prohibited answer elements. The team measures grounding, domain completeness, correct refusal when evidence is missing and time to draft. The same suite runs before every model or prompt change, and failures found during the pilot are added as new cases.
What to remember
Define quality from the perspective of the actual work. A small, maintained test set containing real cases and explicit failure limits is more useful for a business decision than a general model benchmark.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed