The control pattern
Data asset
Changing production inputs and relationships
Knobs
Monitoring window, Segment, Baseline, Alert threshold, Seasonality, Retraining trigger
Outcome
Early detection of changing conditions
Measurement
See how to measure this below
What it is
Data drift occurs when production inputs differ from the data used to build or validate a system. Concept drift occurs when the relationship between those inputs and the desired outcome changes. The incoming data stream is the data asset; the comparison window, baseline, segment, threshold and retraining rule are the knobs.
Financial markets make drift normal rather than exceptional. Volatility regimes change, sectors rotate, disclosure language evolves, accounting standards shift, and data providers alter schemas.
An alert should not automatically trigger retraining. First determine whether the cause is a data defect, a temporary event, a source change or a durable regime shift.
The knobs in detail
Each row is one adjustable property of the data asset, and what moving it tends to do.
| Knob | What you adjust | Likely effect |
|---|---|---|
| Monitoring window | How much history the comparison uses | Trades sensitivity against noise |
| Segment | Which slices are watched separately | A stable average hides concentrated drift |
| Baseline | What current data is compared against | Defines what counts as normal |
| Alert threshold | How much deviation triggers attention | Balances vigilance against alert fatigue |
| Seasonality | Expected periodic variation | Prevents earnings season looking like drift |
| Retraining trigger | What actually causes a rebuild | Keeps response proportionate to cause |
Applied: Stocks Assistant
Stocks Assistant should monitor both technical and market drift. Technical indicators include missing fields, changed document layouts, unusual token lengths, OCR degradation and shifts in feed latency. Behavioural indicators include changing question types and lower retrieval success. Market indicators include changes in volatility, correlations, liquidity, earnings-surprise distributions and the performance of ranking factors. Segment-level monitoring is essential because a stable overall average can hide drift concentrated in one sector or document type.
How to measure it
Evidence that the knob produced the intended behaviour, rather than shifting the problem elsewhere.
- Time to detection
- Affected coverage
- False-alert rate
- Post-response recovery
- A versioned record of every intervention
Common mistakes
Retraining on every alert instead of diagnosing the cause first.
Monitoring only the overall average, which hides sector- or document-level drift.