Embedding models are easy to compare badly.
A leaderboard says Model A scores 67.2 while Model B scores 66.8.
Model A must be better.
Not necessarily.
Your production workload may contain legal documents, product SKUs, multilingual questions, long passages, tables, abbreviations, customer terminology, hard negatives and metadata filters that a benchmark barely represents.
Does this embedding model help our application retrieve the right information, at the required quality, latency and cost, on our actual data?
The practical evaluation hierarchy is:
Public benchmark → Domain benchmark → Retrieval evaluation → End-to-end RAG evaluation → Production monitoring
What is embedding evaluation?
Embedding evaluation determines how well vector representations preserve relationships that matter for a specific task.
Tasks include semantic search, RAG retrieval, recommendation, duplicate detection, clustering, classification, entity matching, multimodal search and long-term agent memory.
Embedding quality is task-dependent.
Start with the downstream task
Semantic search may prioritize Recall@K, MRR and NDCG.
Classification may prioritize accuracy and F1.
Recommendation may require click-through, conversion or engagement.
The metric should follow the job.
Intrinsic vs extrinsic evaluation
Intrinsic evaluation inspects semantic similarity, clustering and vector neighborhoods.
Extrinsic evaluation measures actual downstream task performance.
For production systems, extrinsic evaluation is usually more important.
Evaluation stack
| Layer | Question |
|---|---|
| Public benchmark | Is this model broadly capable? |
| Domain benchmark | Does it understand our terminology? |
| Retrieval/application test | Does it improve our system? |
| Production test | Does it work in reality? |
Public benchmarks shortlist candidates.
Your domain benchmark should make the selection.
MTEB
MTEB is a useful benchmark ecosystem for embedding and retrieval systems across many tasks, languages and domains.
Do not interpret it as:
Top model = correct production choice.
Use it to identify models worth testing internally.
Retrieval metrics
Core metrics include:
- Recall@K,
- Precision@K,
- MRR,
- NDCG,
- Hit Rate,
- MAP.
Recall@K
Did the correct evidence appear in the top K results?
For RAG, missing evidence cannot be recovered by generation.
Precision@K
How much retrieved material is actually relevant?
Too much irrelevant context consumes tokens and can confuse downstream generation.
MRR
How early does the first relevant result appear?
NDCG
How well are graded-relevance results ranked near the top?
Build a domain-specific dataset
A useful evaluation record contains:
- query,
- relevant documents,
- hard negatives,
- language,
- category,
- risk.
Use real search queries, support questions, SME questions, failed RAG queries and edge cases.
Hard negatives are critical
Use plausible but wrong documents:
- old policy versions,
- similar products,
- wrong jurisdictions,
- neighboring regulations,
- another tenant’s record.
These reveal whether the embedding captures the distinction that matters.
Query-document asymmetry matters
Some models encode queries and documents differently.
Benchmark using the intended production instruction or input type.
Test exact search first
Use exact vector search where practical to isolate embedding quality.
Then introduce the production ANN index so you can separate representation failure from index failure.
Evaluate the real retrieval pipeline
A useful progression is:
Embedding → Exact Search → ANN Search → Metadata Filters → Hybrid Retrieval → Reranking → RAG
Measure each stage.
Evaluate metadata filtering
Benchmark under realistic tenant, jurisdiction, status, date and permission filters.
Evaluate hybrid retrieval
Dense retrieval handles semantics.
Sparse/keyword retrieval handles exact identifiers well.
Compare dense, BM25 and hybrid search.
Evaluate reranking separately
If you retrieve Top-50 then rerank to Top-5, measure candidate recall before reranking and final ranking after reranking.
Separate embedding and generation quality
Correct evidence can still produce a bad answer.
Wrong evidence can produce a faithful but incorrect answer.
Maintain separate retrieval and generation metrics.
Embedding metric hierarchy for RAG
Embedding Recall
↓
Ranking Quality
↓
Context Quality
↓
Groundedness
↓
Answer Correctness
↓
Business Outcome
Multilingual evaluation
Slice by language, script, region and domain.
Also test cross-lingual query/document pairs if the application needs them.
Domain vocabulary
Build queries around actual business terminology, abbreviations, product names, regulations and specialized concepts.
Numbers, dates and identifiers
Dense models can treat near-identical numeric passages as semantically equivalent.
Create evaluation slices for numbers, dates, product IDs, names and versions.
Temporal retrieval
Test current, historical and future-effective documents.
This reveals whether the retrieval stack finds the correct version.
Dimension size is a knob
Higher dimensionality is not automatically better.
Larger vectors increase storage, bandwidth and index cost.
Test the smallest representation that satisfies quality requirements.
Compression and quantization
Compare full precision with compressed vectors.
Measure quality lost per byte or dollar saved.
Model size is not quality
A larger model may cost more and reduce throughput.
Compare quality gain against production overhead.
Latency and throughput
Measure query embedding latency, document embedding throughput, retrieval latency and re-embedding throughput.
Embedding cost
Include initial corpus embedding, incremental updates, query embeddings, model migrations and infrastructure.
Large-corpus re-embedding can be a major migration cost.
Versioning
Track embedding model, version, dimensions, distance metric and chunk strategy.
This is essential for migration and reproducibility.
Do not mix incompatible spaces
Queries and documents should generally use compatible encoders.
Model migration may require re-embedding the corpus.
Critical-query test sets
High-risk queries such as regulated disclosures or safety instructions may need stricter recall targets than ordinary queries.
Fairness and privacy
Embedding spaces can encode unwanted bias and sensitive information.
Apply suitable governance to sensitive embedding indexes.
Multimodal embeddings
For multimodal systems, test text→image, image→text, image→image and document-page retrieval separately.
Agent memory
Long-horizon agent memory is not identical to ordinary document retrieval.
Test distant events, temporal ordering, user preferences and procedural memory.
Public benchmarks should shortlist; private benchmarks should decide
Public benchmark
↓
shortlist 3–5 models
↓
enterprise benchmark
↓
cost + latency benchmark
↓
RAG / application evaluation
↓
production test
Build an embedding scorecard
Include:
- Recall@5,
- Recall@10,
- NDCG@10,
- critical-query recall,
- P95 latency,
- re-embedding throughput,
- index size,
- grounded answer rate,
- cost.
Slice the scorecard by document type, language, domain, query class, freshness, identifiers and high-risk queries.
Controlled experiments
Do not change model and chunk size simultaneously unless testing their interaction.
Controlled experiments reveal causality.
Treat embedding configuration as KNOBS
Important variables include:
- embedding model,
- dimensions,
- query/document instructions,
- chunk size,
- overlap,
- distance metric,
- ANN algorithm,
- ANN parameters,
- Top-K,
- hybrid weight,
- reranker.
Measure impact on recall, precision, latency, index size, cost and RAG quality.
Bottom line
Do not ask only:
Which embedding model has the highest benchmark score?
Ask:
Which embedding configuration retrieves the right evidence for our actual workload, at the required quality, latency, scale and cost?
Use public benchmarks to shortlist, private benchmarks to decide, retrieval metrics to diagnose and end-to-end RAG outcomes to validate the final choice.
Frequently asked questions
How do you evaluate an embedding model?
Use representative queries and relevant documents, measure Recall@K, MRR and NDCG, then validate downstream application quality, latency, storage and cost.
What is MTEB?
An open benchmark ecosystem for comparing embedding and retrieval systems across many tasks, languages and domains.
Is the highest-ranked MTEB model always best?
No. Public benchmarks should shortlist models; your own domain data and workload should determine production selection.
