The control pattern
Data asset
Source records and canonical data models
Knobs
Completeness, Validity, Consistency, Identity resolution, Units and adjustments, Lineage and deduplication
Outcome
Dependable downstream analysis
Measurement
See how to measure this below
What it is
Data quality is the foundation beneath every model and score. Source records are the data asset; schema rules and validation properties are the knobs. Completeness asks whether required fields exist, validity checks permitted values, consistency compares related records, and deduplication prevents the same event appearing multiple times.
Financial systems also need careful identity resolution, unit normalisation, currency handling and treatment of corporate actions. A sophisticated model cannot reliably repair a pipeline that confuses millions with billions or maps a ticker to the wrong legal entity.
In practice, improving these knobs often produces larger gains than switching to a more capable language model.
The knobs in detail
Each row is one adjustable property of the data asset, and what moving it tends to do.
| Knob | What you adjust | Likely effect |
|---|---|---|
| Completeness | Whether required fields exist | Determines what can be computed at all |
| Validity | Whether values are permitted | Catches malformed records at the door |
| Consistency | Whether related records agree | Surfaces reconciliation failures |
| Identity resolution | Mapping symbols to legal entities | Prevents the wrong company entirely |
| Units and adjustments | Currency, scale, splits, restatements | Stops silent order-of-magnitude errors |
| Lineage and deduplication | Where a value came from, and how often | Makes every score traceable |
Applied: Stocks Assistant
For Stocks Assistant, create canonical entities for company, security, filing, fiscal period, market observation, analyst estimate, transcript segment, options contract and portfolio position. Preserve both reported and normalised values along with the transformation used. Distinguish adjusted and unadjusted prices; record splits, dividends, symbol changes and restatements; and use nulls rather than silently converting missing data to zero. Each derived score should be traceable to source records and a versioned transformation.
How to measure it
Evidence that the knob produced the intended behaviour, rather than shifting the problem elsewhere.
- Feed latency and stale-record counts
- Missing fields and duplicate rate
- Reconciliation differences between related sources
- Unit anomalies and lineage gaps
- Quality gates passed before data enters RAG indexes, training, scoring or reports
Common mistakes
Serving plausible-looking values from an unknown state when a source fails. Degrade visibly and selectively instead.
Converting missing data to zero, which turns absence into a false fact.