The control pattern
Data asset
Risk examples, policies and runtime checks
Knobs
Risk categories, Coverage, Thresholds, Adversarial cases, Permissions, Escalation rules
Outcome
Safer outputs and actions
Measurement
See how to measure this below
What it is
Safety guardrails define what the system should refuse, qualify, verify or escalate. The data asset includes risk-labelled examples, explicit policies, deterministic checks and runtime signals.
Knobs include which risk categories are covered, how conservative the thresholds are, what actions require confirmation, and how adversarial or manipulative inputs are handled. Guardrails should be layered: a prompt alone is too fragile, while a rigid rule set alone cannot interpret every context. Combining policy-aware models with structured validation and permissions produces a more dependable boundary.
Overly permissive systems create harm; overly restrictive systems drive users to ignore warnings. The best guardrail explains the boundary in plain language and redirects toward verifiable information.
The knobs in detail
Each row is one adjustable property of the data asset, and what moving it tends to do.
| Knob | What you adjust | Likely effect |
|---|---|---|
| Risk categories | Which harms are explicitly covered | Defines the boundary being enforced |
| Coverage | How much of the surface is checked | Closes gaps between policy and product |
| Thresholds | How conservative each check is | Balances harm against false refusals |
| Adversarial cases | Manipulative and injected inputs | Hardens the system against source text |
| Permissions | What requires explicit confirmation | Keeps consequential actions deliberate |
| Escalation rules | When a human or larger model steps in | Provides a safe path instead of a refusal |
Applied: Stocks Assistant
For Stocks Assistant, safety controls should prevent fabricated certainty, unqualified guarantees, unsupported price targets, and claims that a signal is personalised advice when suitability data is missing. Retrieved documents can contain malicious or irrelevant instructions, so the system should treat source text as evidence rather than executable commands. A research assistant should be able to explain risk, but direct trade execution or account changes should stay outside the workflow unless separately authorised with clear confirmation and auditability. High-volatility products and options deserve stronger warnings than general educational content.
How to measure it
Evidence that the knob produced the intended behaviour, rather than shifting the problem elsewhere.
- Policy-violation rate on ordinary and adversarial tests
- False refusal rate
- Escalation quality : is the safe alternative still useful?
- Auditability of any consequential action
Common mistakes
Relying on prompt instructions alone as the safety layer.
Treating retrieved document text as instructions rather than evidence.