The control pattern
Data asset
Text, tables, charts, images and document layout
Knobs
Resolution, OCR engine, Page selection, Layout preservation, Modality balance, Cross-modal validation
Outcome
Better understanding of complex financial documents
Measurement
See how to measure this below
What it is
Multimodal AI works across text, images, tables, charts and layout. The source document is the data asset, while resolution, OCR quality, page selection, modality balance and layout preservation are knobs.
Many financial facts lose meaning when reduced to plain text: a table header may define the period for several columns, indentation may distinguish a subtotal, and a chart may reveal a trend not stated in the surrounding narrative. Treating layout as data helps the system preserve these relationships.
The goal is not merely to describe an image, but to recover financially meaningful structure without losing the context that makes a number interpretable.
The knobs in detail
Each row is one adjustable property of the data asset, and what moving it tends to do.
| Knob | What you adjust | Likely effect |
|---|---|---|
| Resolution | How finely pages are rendered | Trades fidelity against cost and latency |
| OCR engine | Which extractor reads the page | Sets the floor on character accuracy |
| Page selection | Which pages get visual processing | Keeps cost proportionate to value |
| Layout preservation | Whether structure survives extraction | Retains period headers and subtotals |
| Modality balance | Text-native vs. vision paths | Routes each page to the right pipeline |
| Cross-modal validation | Reconciling OCR with table arithmetic | Disagreement becomes an uncertainty signal |
Applied: Stocks Assistant
For Stocks Assistant, a multimodal pipeline can extract tables from filings, read investor-presentation charts, interpret scanned annual reports, and connect transcript statements with displayed guidance. Use high-resolution rendering only where needed, because processing every page visually adds cost and latency. Route text-native pages through structured extraction, and use vision on tables, charts or low-quality scans. Retain page and bounding-box provenance so a user can inspect the exact region behind a result.
How to measure it
Evidence that the knob produced the intended behaviour, rather than shifting the problem elsewhere.
- Cell accuracy, and row/column association
- Unit recognition
- Chart-value tolerance
- OCR character error rate
- Evidence localisation : can the user find the exact region?
Common mistakes
Rendering every page at high resolution, which buys accuracy you did not need at a cost you did not budget.
Flattening tables into prose, which destroys the header relationships that define the period.