The control pattern
Data asset
Test datasets and scoring rules
Knobs
Scenario mix, Difficulty, Time split, Segment coverage, Scoring method, Pass thresholds
Outcome
Reliable measurement of system quality
Measurement
See how to measure this below
What it is
Evaluation data determines what the organisation can see, and therefore what it can improve. A test set is a control surface because teams naturally optimise toward the failures and metrics it exposes.
Its knobs include scenario mix, difficulty, coverage of rare conditions, scoring method, reviewer rubric, time window and release threshold. A single average score is rarely sufficient: it can hide severe weaknesses in certain sectors, document types, market regimes or user groups. Evaluation should be sliced by the properties that matter operationally.
Evaluation becomes a true control knob when a model, prompt, retriever or scoring change cannot ship unless it improves the intended metric without crossing a guardrail elsewhere.
The knobs in detail
Each row is one adjustable property of the data asset, and what moving it tends to do.
| Knob | What you adjust | Likely effect |
|---|---|---|
| Scenario mix | Which tasks and cases are represented | Determines what failures are visible at all |
| Difficulty | Share of hard and rare conditions | Prevents easy averages from hiding weakness |
| Time split | Point-in-time boundaries on evidence | Stops future information leaking into tests |
| Segment coverage | Sectors, document types, user groups | Exposes concentrated failure |
| Scoring method | Metrics and reviewer rubric | Defines what counts as correct |
| Pass thresholds | Bar for release, by risk level | Turns measurement into a gate |
Applied: Stocks Assistant
Stocks Assistant should maintain separate evaluation suites for factual extraction, earnings summarisation, sentiment interpretation, ranking, portfolio explanation, options analysis and conversational question answering. Financial evaluations must be point-in-time: a model answering a historical question may use only information that was available at that time, or revised filings, later analyst estimates and future prices leak into the test. Ranking evaluations should include transaction assumptions and survivorship-bias controls.
How to measure it
Evidence that the knob produced the intended behaviour, rather than shifting the problem elsewhere.
- Numeric accuracy and citation precision
- Unsupported claims and completeness
- Confidence calibration
- Latency and cost
- A frozen core benchmark for comparability, plus a rolling set of new production failures
Common mistakes
Reporting one average score. It hides exactly the weaknesses that matter operationally.
Applying the same pass threshold to high-risk and low-risk tasks. One threshold for everything protects nothing.