The control pattern
Data asset
Training examples
Knobs
Example quality, Volume, Task mix, Difficulty, Weighting, Curriculum order
Outcome
Specialised and repeatable model behaviour
Measurement
See how to measure this below
What it is
Fine-tuning turns a general model into a specialist by repeatedly showing it the inputs and outputs that define the desired behaviour. The training examples are the data asset you change: changing them changes what the model learns to notice, how it responds, and which mistakes it avoids.
The knobs are the properties of those examples. Quality controls whether the target answer is correct. Quantity controls how often the model sees a pattern. Task mix determines which capabilities receive capacity. Weighting emphasises high-value or high-risk situations. Curriculum order can move the model from straightforward cases toward difficult edge cases.
Fine-tuning is closely related to distillation when the examples come from a larger teacher model, but the two are not identical. Fine-tuning describes the adaptation process; distillation describes transferring behaviour from a teacher into a smaller student.
The knobs in detail
Each row is one adjustable property of the data asset, and what moving it tends to do.
| Knob | What you adjust | Likely effect |
|---|---|---|
| Example quality | Whether the target answer is actually correct | Sets the ceiling on everything learned |
| Volume | How often the model sees a given pattern | Controls how firmly a behaviour sticks |
| Task mix | Extraction vs. classification vs. explanation | Decides which capabilities get capacity |
| Difficulty | Clean cases vs. ambiguous ones | Trades routine speed against edge-case reasoning |
| Weighting | Loss weight or sampling frequency | Emphasises high-value or high-risk situations |
| Curriculum order | Simple examples first, hard ones later | Shapes how the model generalises to the tail |
Applied: Stocks Assistant
For Stocks Assistant, fine-tuning could specialise a compact model in extracting revenue, EPS, margin changes, guidance, risks and management commentary from earnings materials. Examples should include both clean and difficult cases: companies with non-standard fiscal calendars, restated metrics, non-GAAP reconciliations, missing guidance, conflicting numbers, and call transcripts where the Q&A tone differs from prepared remarks. High-impact fields should carry more weight than stylistic preferences : a factual error in reported EPS matters more than whether a summary uses three bullets or four. The model should also learn to abstain on missing or ambiguous evidence rather than being rewarded for always producing an answer.
How to measure it
Evidence that the knob produced the intended behaviour, rather than shifting the problem elsewhere.
- Field-level accuracy on the metrics that matter most
- Unsupported-claim rate and citation alignment
- Performance across sectors, not just in aggregate
- Calibration of stated uncertainty against observed error
- A frozen test set that never enters training
- Comparison against both the teacher and a simple rules baseline
Common mistakes
Mistaking fluency for accuracy : a successful fine-tune is more correct, consistent and predictable, not merely smoother.
Rewarding an answer for every input, which trains away the abstention behaviour you need most on hard documents.