In brief
- A public benchmark does not replace evaluation with the organisation's own cases and languages.
- Model quality includes correctness, grounding, format, latency, robustness and cost.
- Provider and model are separate decisions because the same model may be available through different deployment routes.
- Selection criteria, test data, versions and accepted trade-offs should be documented for traceability.
Why does comparison start with the task?
Models perform differently across languages, document types, tool use and output formats. The general question 'Which model is best?' therefore has no dependable answer. First define what the application must do and which failures are unacceptable.
A test set should include typical cases, difficult exceptions, incomplete input and tasks the system should refuse or escalate. Every candidate receives the same application context and is scored with the same method.
- Cover internal terminology and the languages used in practice
- Include normal, edge, risk and unanswerable cases
- Version model, prompt, sources and tool configuration
- Set minimum thresholds and critical exclusions before testing
Which quality dimensions should be measured?
One average score can hide important weaknesses. Measure separately whether the solution is correct, complete, grounded and compliant with the required format. Production use also depends on latency, stability and behaviour when context is missing.
The case count, thresholds and weights below form one coherent document-extraction example and are not universal standards. Set them before testing to match the process, risk, language, volume and acceptable failures; the model weights in this example add up to 100 percent.
| Model criterion | Measurement | Illustrative minimum | Weight | Evidence |
|---|---|---|---|---|
| Domain task success | Blind review of 180 representative cases | At least 97% correct mandatory fields; no critical failure | 35 % | Versioned evaluation report with failure categories |
| Grounding and refusal | Check citations and deliberately unanswerable cases | At least 95% correct citations and 98% correct escalations | 25 % | Evidence and refusal rationale for each test case |
| Format and tool use | Schema validation and permitted tool paths | 99% valid format; no prohibited action | 15 % | Automated test records and tool logs |
| Robustness | Repeat paraphrases, missing inputs and disruptions | Quality loss no greater than 5 percentage points | 10 % | Regression suite across variants and model versions |
| Latency and cost | End-to-end measurement per successful case | 95th percentile below 8 seconds and under CHF 0.20 | 15 % | Measurements from an identical test environment |
What should be assessed beyond the model?
The provider determines the regions and deployment modes in which a model is available, how input is processed and stored, and which service commitments apply. Support, change notices and access to logs also affect operations.
Keep model and provider assessments as two separate results. After mandatory gates have passed, the two weighted provider criteria in this example form their own 100-percent provider score; any later overall decision must show both scores and every exclusion explicitly.
| Provider criterion | Assessment | Gate or provider weight | Required evidence |
|---|---|---|---|
| Data terms | Match the exact service, plan, region, retention, training, deletion and subprocessors | Mandatory gate | Contract, service description and documented data flow agree |
| Identity and security | Test SSO, roles, network, encryption, logs and incident notification | Mandatory gate | Technical pilot plus current security and audit evidence |
| Operations and change | Simulate quotas, availability, model versions, retirement and a support case | 55% of the separate provider score | SLA, change notice and recorded support test |
| Commercial and exit | Calculate price tiers, commitments, liability, export and termination effects | 45% of the separate provider score | Three-year cost model and successful export test |
How does the selection remain traceable?
Weight the criteria before making a decision and ask business, IT, security, privacy and operations to define their minimum requirements. A candidate that violates an exclusion criterion should not be rescued by a high average score.
- Use a shortlist of plausible candidates rather than an unfocused market survey
- Score responses blindly or at least under identical conditions
- Assess model, deployment route and provider separately
- Retain a repeatable test for later model and version changes
Example: extracting data from technical service reports
Three models extract error code, component and action from 180 anonymised reports in German, French and English. The team measures field accuracy, correct blanks where information is absent, latency and cost. Two providers meet the quality bar. The final choice is the deployment route that supports existing identities, the required region and version control. The test set remains as a regression suite.
What to remember
Compare candidates on the same real tasks and explicit exclusion limits. Then decide on the complete deployment, not on a leaderboard position or model name.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed