Mine and Refine: Optimizing Graded Relevance in E-commerce Search Retrieval

Paper: 2602.17654 Authors: Jiaqi Xi, Raghav Saboo, Luming Chen, Martin Wang, Sudeep Das Categories: cs.IR, cs.LG

Abstract

This paper introduces a two-stage “Mine and Refine” framework for training semantic embeddings in large-scale e-commerce search systems. The approach addresses the challenge of graded relevance, where search results can be exact matches, substitutes, or complements. Stage 1 trains a multilingual Siamese two-tower retriever using label-aware supervised contrastive learning to establish a robust semantic space. Stage 2 mines hard negative samples via approximate nearest neighbor search, re-annotates them using a policy-aligned LLM, and applies a multi-class extension of circle loss to sharpen boundaries between relevance levels. The framework incorporates spelling augmentation and synthetic query generation for robustness, demonstrating significant improvements in both offline metrics and production A/B tests.

Key Contributions

• A two-stage training framework that explicitly optimizes for graded relevance in e-commerce search retrieval
• Policy-consistent supervision at scale through fine-tuning a lightweight LLM on human annotations with three-level relevance guidelines
• Multi-class extension of circle loss that sharpens similarity score boundaries between different relevance strata
• Hard sample mining strategy using ANN followed by LLM-based re-annotation to refine the embedding space
• Robustness improvements through additive spelling augmentation and synthetic query generation
• Demonstrated production impact with statistically significant gains in engagement and business metrics

Methodology: Two-Stage Training Pipeline

The framework operates in two distinct stages, each addressing specific challenges in e-commerce search retrieval.

Stage 1: Global Semantic Space Construction

The first stage establishes a robust multilingual semantic space using a Siamese two-tower architecture. The model employs a label-aware supervised contrastive objective that considers the graded nature of relevance. Unlike binary relevance approaches, this objective function explicitly accounts for three relevance levels: exact matches, substitutes/complements, and irrelevant items.

The training data is obtained through a carefully designed supervision pipeline. A lightweight LLM is fine-tuned on human annotations following a three-level relevance guideline that aligns with product policies and business constraints. This ensures consistency and scalability while maintaining quality. The supervision is further refined through engagement-driven auditing, where user interaction signals help identify and correct residual annotation noise.

Stage 2: Hard Sample Mining and Boundary Refinement

The second stage focuses on refining the embedding space by explicitly sharpening the boundaries between relevance levels. Hard samples are mined using approximate nearest neighbor (ANN) search, identifying cases where the model’s current embeddings place items of different relevance levels too close together in the semantic space.

These hard samples are re-annotated using the policy-aligned LLM to ensure label quality. The key innovation is a multi-class extension of circle loss, which explicitly optimizes for clear separation between the three relevance strata. The loss function can be expressed as:

$\mathcal{L} = \log[1 + \sum_{i \in \Omega_p} \sum_{j \in \Omega_n} \exp(\gamma(s_j - s_i + m))]$

where $s_i$ and $s_j$ are similarity scores for positive and negative pairs, $\gamma$ is a scale factor, and $m$ is a margin parameter. The multi-class extension ensures that exact matches have higher similarity scores than substitutes, which in turn score higher than irrelevant items.

Robustness and Generalization

E-commerce search faces unique challenges including long-tail queries, spelling variations, and noisy user inputs. The framework addresses these through two complementary strategies:

Additive Spelling Augmentation: During training, the system introduces controlled spelling variations to queries, helping the model learn robust representations that generalize to misspelled or noisy inputs. This is particularly important for e-commerce where users often type quickly on mobile devices.

Synthetic Query Generation: To improve coverage of the long tail, the framework generates synthetic queries for products using language models. This augmentation ensures the retriever can handle diverse query formulations and rare product categories that may have limited organic training data.

The multilingual nature of the two-tower architecture also enables the system to generalize across languages, which is essential for global e-commerce platforms serving diverse markets.

Experimental Results and Production Impact

The paper reports comprehensive evaluation across both offline metrics and production A/B tests. Offline evaluations demonstrate improvements in standard retrieval metrics including precision, recall, and normalized discounted cumulative gain (NDCG) across all three relevance levels.

The multi-class circle loss shows particular effectiveness in creating clear separation between relevance strata. Analysis of similarity score distributions reveals that the refined embeddings produce distinct clusters for exact matches, substitutes, and irrelevant items, enabling more stable hybrid ranking and clearer thresholding in production systems.

Production A/B tests validate the offline improvements, showing statistically significant gains in user engagement metrics including click-through rate, add-to-cart rate, and conversion rate. The business impact extends beyond engagement, with measurable improvements in revenue metrics. The clear separation of relevance levels also improves system stability, reducing the need for manual threshold tuning and making the retrieval system more maintainable.

Takeaways

1. Graded relevance is a critical consideration for e-commerce search systems, where users often accept substitutes and complements beyond exact matches
2. Two-stage training with hard sample mining and refinement can significantly improve embedding quality compared to single-stage approaches
3. Policy-aligned LLM supervision provides a scalable path to high-quality training data that respects business constraints
4. Multi-class extensions of contrastive losses can explicitly optimize for clear separation between multiple relevance levels
5. Robustness techniques including spelling augmentation and synthetic query generation are essential for handling real-world e-commerce search challenges
6. Clear separation of similarity scores across relevance strata improves not just retrieval quality but also system stability and maintainability in production