How to Benchmark Embedding Models On Your Own Data

This comprehensive freeCodeCamp course teaches you how to properly evaluate embedding models for your specific data and use case, rather than relying solely on public benchmarks.

Why Benchmark Your Own Data?

Public benchmarks (MTEB, etc.) show general performance, but:

• Your data is unique
• Your queries are specific
• Domain matters (legal, medical, technical)
• What works generally may not work for you

Understanding Embeddings

What Are Embeddings?

• Dense vector representations of text
• Capture semantic meaning
• Enable similarity search
• Typical dimensions: 384, 768, 1024, 1536

How They Work

1. Text input
2. Model processes through layers
3. Output: fixed-size vector
4. Similar meanings → similar vectors

Key Properties

• Dimensionality: Vector size (affects storage, speed)
• Semantic capture: How well meaning is preserved
• Domain adaptation: Performance on specific domains

Popular Embedding Models

Open Source

• all-MiniLM-L6-v2: Fast, 384 dimensions
• all-mpnet-base-v2: Better quality, 768 dimensions
• BGE models: Strong multilingual support
• E5 models: Instruction-tuned embeddings

Commercial

• OpenAI text-embedding-3: Small and large variants
• Cohere embed: Multiple sizes
• Voyage AI: Domain-specific options

Benchmarking Methodology

Step 1: Create Evaluation Dataset

Build a dataset that reflects your actual use case:

evaluation_data = [
    {
        "query": "What is the refund policy?",
        "relevant_docs": ["doc_123", "doc_456"],
        "irrelevant_docs": ["doc_789"]
    },
    # ... more examples
]

Step 2: Define Metrics

Retrieval Metrics:

• Recall@K: % of relevant docs in top K
• Precision@K: % of top K that are relevant
• MRR (Mean Reciprocal Rank): Position of first relevant result
• NDCG: Normalized Discounted Cumulative Gain

Similarity Metrics:

• Cosine similarity
• Dot product
• Euclidean distance

Step 3: Run Benchmarks

def benchmark_model(model, eval_data):
    results = []
    for item in eval_data:
        query_embedding = model.encode(item["query"])
        doc_embeddings = model.encode(item["docs"])
        similarities = cosine_similarity(query_embedding, doc_embeddings)
        # Calculate metrics
        recall = calculate_recall(similarities, item["relevant_docs"])
        results.append(recall)
    return np.mean(results)

Step 4: Compare Models

Test multiple models on same data:

• Same queries
• Same documents
• Same metrics
• Statistical significance

Practical Considerations

Speed vs Quality

• Smaller models: faster, less accurate
• Larger models: slower, more accurate
• Find your sweet spot

Dimensionality Trade-offs

• Higher dimensions: more information, more storage
• Lower dimensions: faster search, less nuance
• Consider your scale

Domain Adaptation

• General models may miss domain terms
• Consider fine-tuning
• Or use domain-specific models

Building Your Benchmark Suite

1. Query Types

• Short queries (keywords)
• Long queries (questions)
• Complex queries (multi-part)

2. Document Types

• Short snippets
• Long documents
• Technical content
• Conversational text

3. Edge Cases

• Synonyms
• Negations
• Out-of-domain queries
• Multilingual content

Evaluation Framework

class EmbeddingBenchmark:
    def __init__(self, models, eval_data):
        self.models = models
        self.eval_data = eval_data

    def run(self):
        results = {}
        for model_name, model in self.models.items():
            results[model_name] = {
                "recall@5": self.calculate_recall(model, k=5),
                "recall@10": self.calculate_recall(model, k=10),
                "mrr": self.calculate_mrr(model),
                "latency": self.measure_latency(model)
            }
        return results

Best Practices

1. Use representative data - Real queries, real documents
2. Test at scale - Performance may change with volume
3. Consider latency - Speed matters in production
4. Version your benchmarks - Track changes over time
5. A/B test in production - Real users, real feedback

Common Pitfalls

• Overfitting to benchmark data
• Ignoring latency requirements
• Not testing edge cases
• Using wrong similarity metric
• Insufficient test data

Tools & Resources

• sentence-transformers: Easy model loading
• MTEB: Public benchmark reference
• Hugging Face: Model hub
• LangChain: Evaluation utilities