The control pattern
Data asset
Selected production examples for human labelling
Knobs
Uncertainty threshold, Disagreement, Sampling method, Business impact, Novelty, Review priority
Outcome
Faster improvement with fewer labelled examples
Measurement
See how to measure this below
What it is
Active learning chooses which production examples deserve human review instead of labelling data at random. The data asset is the stream of uncertain, novel or high-impact cases; the knobs determine how examples are selected.
Common signals include low confidence, disagreement between models, failure of a validator, unusual input structure, user correction and high business consequence. The method is most effective when the labelling budget is limited and the current model already handles ordinary cases well.
Reviewed examples should feed several destinations: a corrected production record, the evaluation suite, retrieval rules, prompt examples, and eventually fine-tuning or distillation data.
The knobs in detail
Each row is one adjustable property of the data asset, and what moving it tends to do.
| Knob | What you adjust | Likely effect |
|---|---|---|
| Uncertainty threshold | How unsure a case must be to queue | Focuses review where the model is weakest |
| Disagreement | Conflict between models or validators | Surfaces silent errors |
| Sampling method | How the queue is composed | Determines what the team ever sees |
| Business impact | Financial materiality of the case | Prioritises consequential errors |
| Novelty | How unlike the training set the input is | Catches emerging document types |
| Review priority | Order of the queue | Shortens time to fix |
Applied: Stocks Assistant
For Stocks Assistant, a review queue might prioritise conflicting EPS values across sources, a newly introduced filing table, ambiguous ticker resolution, an earnings summary rejected by a user, or a high-confidence recommendation contradicted by a verifier. Examples can be scored on uncertainty, novelty, expected learning value and financial materiality.
How to measure it
Evidence that the knob produced the intended behaviour, rather than shifting the problem elsewhere.
- Label turnaround time
- Reviewer agreement
- Error reduction per labelled example
- Recurrence rate of the targeted failure
Common mistakes
Queueing only known weaknesses. Reserve space for random cases, or you will miss silent failures where the model is confidently wrong.
Retraining on unverified feedback rather than reviewed corrections.