Build GenAI around the task, context and controls: not around a model demo.
A practical guide for choosing implementation patterns, designing evaluation, managing risk, and moving from generative features to bounded agentic workflows.
Start here
Scope the task before selecting technology
“Add AI” is not a product requirement. Describe the human workflow, its bottleneck, the decision boundary, and the evidence required for success.
Outcome
Define the successful user or business result: not merely the generated output.
Context
List source data, freshness, permissions, tools and business rules the system needs.
Consequence
Identify what happens if the answer or action is wrong, late, unsupported or unauthorized.
Evaluation
Build representative cases, expected evidence and a failure taxonomy before optimizing prompts.
Implementation patterns
API, RAG, fine-tuning and self-hosting solve different problems
These are composable architecture choices, not a maturity ladder. Use the minimum complexity that satisfies the workflow.
| Pattern | Best fit | What it gives you | What you still own |
|---|---|---|---|
| Hosted model API | Fast product iteration; broad reasoning/generation | Low model-serving burden and rapid access to capable models | Evaluation, data handling, prompt/context design, cost and vendor dependency |
| RAG / context retrieval | Answers must use current or proprietary knowledge | Grounding in selected enterprise sources with evidence and freshness | Chunking, retrieval quality, permissions, citations, index lifecycle and failure analysis |
| Fine-tuning / adaptation | Systematic behavior, format, task or domain adaptation | More consistent model behavior for the trained objective | Training data quality, evaluation, retraining, drift and version management |
| Self-hosted / open-weight | Deployment control, privacy boundary, customization or scale economics | Control over runtime and infrastructure | Serving, security, capacity, optimization, upgrades and incident response |
| Train a foundation model | Rare cases with strategic data, extreme scale and deep research capability | Maximum control over weights and training process | Data, compute, research, alignment, safety, evaluation and long-term model operations |
RAG and fine-tuning are not substitutes.
RAG supplies external context at inference time. Fine-tuning changes model behavior or representations through training. A production system may use both: or neither: depending on the problem.
Modalities
Design around the information the workflow actually uses
Text is only one modality. Documents, images, audio, video, code and structured records often belong in the same workflow.
Language + documents
Generation, summarization, extraction, classification, translation, document Q&A and reasoning over enterprise text.
Image + visual
Image understanding, document vision, quality inspection, visual search, generation and editing.
Voice + audio
Speech recognition, call intelligence, voice agents, transcription, diarization and speech generation.
Video
Scene understanding, indexing, summarization, multimodal retrieval and generated or edited video experiences.
Code + tools
Code generation, review, execution, test generation, structured tool calls and developer workflows.
Structured + operational data
Use LLMs with metrics, catalogs, events, relational data and APIs instead of flattening every business fact into prose.
From generation to action
Autonomy is a spectrum, not a product label
As systems gain tools and action authority, the design problem moves from prompt quality to execution control.
The model produces content. A person decides what to do with it.
The system retrieves enterprise context and cites evidence before generating.
The system recommends decisions or prepares actions; a person approves.
The system uses approved tools for narrow, reversible actions and routes exceptions.
The system plans multi-step work with explicit budgets, permissions, checkpoints, telemetry and rollback.
Failure modes and controls
Replace fake risk scores with observable evidence
A “risk reduction” percentage is meaningless unless it comes from a measured control test. Use a failure taxonomy and verify that each control works.
| Failure mode | Control pattern | Evidence to collect |
|---|---|---|
| Unsupported or false claims | Grounding, claim checks, constrained scope, refusal/fallback | Unsupported-claim rate, citation validity, task-specific error severity |
| Wrong context retrieved | Hybrid retrieval, metadata filters, reranking, access-aware retrieval | Recall/precision, nDCG/MRR, source coverage, permission tests |
| Sensitive-data exposure | Data classification, redaction, policy filters, provider/storage controls | Data-flow inventory, red-team cases, retention and access logs |
| Unsafe action | Least-privilege tools, approvals, idempotency, limits, sandboxing | Tool-call traces, blocked-action tests, approval rate, rollback drills |
| Cost / latency runaway | Routing, model tiers, caching, budgets, timeouts, bounded loops | Cost per successful task, p95 latency, retry/loop distribution |
| Behavior changes after upgrade | Version pinning, regression eval, shadow/canary, rollback | Pre/post eval diff, incident rate, model/version lineage |
For product and engineering managers
A GenAI PRD needs an operating contract
Normal functional requirements are not enough when behavior is probabilistic and the underlying model can change.
Required PRD additions
- •Representative evaluation set and failure taxonomy
- •Fallback and “I don’t know” behavior
- •Human review and approval points
- •Data handling, retention and permission rules
- •Model / prompt / retrieval version-change plan
- •Production monitoring and rollback ownership
Metrics that predict product value
Roll out as Crawl → Walk → Run.
Crawl: AI drafts; humans approve. Walk: high-confidence bounded cases can proceed while exceptions route to people. Run: broader automation is earned through measured performance, monitoring and rollback: not declared because the model is called an “agent.”
FAQ
Implementation questions
Should we use an LLM API, RAG, fine-tuning, or self-hosting?
These choices can be combined. Start with the simplest access method that meets your data boundary, add retrieval when current enterprise context is required, fine-tune when behavior needs systematic adaptation, and self-host when deployment control or economics justify the operating burden.
What belongs in a GenAI PRD?
Add an evaluation set, failure taxonomy, fallback behavior, human review points, data-handling rules, production metrics, model-change plan, audit requirements and rollback path.
What changes when GenAI becomes agentic?
The system gains tools, multi-step execution and action authority. Tool permissions, budgets, checkpoints, observability, idempotency and rollback become first-class product requirements.
What should we optimize first?
Start with task success and the dominant failure mode. Model swaps are only one lever; context quality, retrieval, prompts, tools and workflow design can be bigger knobs.