IntRec: Intent-based Retrieval with Contrastive Refinement

Paper: 2602.17639 Authors: Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, Yue Lu Categories: cs.CV

Abstract

IntRec addresses a fundamental limitation in open-vocabulary object detection: the inability to refine predictions based on user feedback. Traditional detectors operate in a one-shot manner, making them ineffective when queries are ambiguous or scenes contain multiple similar objects. This paper introduces an interactive retrieval framework centered on an Intent State (IS) mechanism that maintains dual memory sets—positive anchors for confirmed cues and negative constraints for rejected hypotheses. Through contrastive alignment, the system ranks candidate objects by maximizing similarity to positive examples while penalizing rejected ones, enabling precise disambiguation in cluttered environments.

Key Contributions

• Introduction of Intent State (IS), a dual-memory mechanism that tracks both positive anchors (user-confirmed objects) and negative constraints (rejected candidates) throughout the interaction loop
• A contrastive alignment function that dynamically ranks objects by balancing similarity to positive cues against dissimilarity from negative examples
• Demonstration of substantial performance gains with minimal computational overhead: +7.9 AP improvement on LVIS-Ambiguous after single feedback with <30ms latency
• State-of-the-art results on LVIS dataset (35.4 AP), outperforming existing methods like OVMR (+2.3), CoDet (+3.7), and CAKE (+0.5)

Methodology and Architecture

The IntRec framework operates through an iterative refinement process. At initialization, the system processes a user query through an open-vocabulary detector to generate initial candidate proposals. The Intent State module then maintains two evolving memory banks: $\mathcal{M}^+$ for positive anchors and $\mathcal{M}^-$ for negative constraints.

The core innovation lies in the contrastive alignment function:

where $c_i$ represents a candidate object, $\text{sim}(\cdot)$ computes feature similarity, and $\alpha, \beta$ are weighting hyperparameters. This formulation explicitly models user intent by simultaneously attracting candidates toward confirmed examples and repelling them from rejected ones.

When users provide feedback—either confirming a correct detection or rejecting an incorrect one—the system updates the respective memory bank and recomputes rankings. This iterative process continues until the user’s target object is successfully retrieved or a maximum interaction limit is reached.

Experimental Results and Analysis

On the LVIS dataset, IntRec achieves 35.4 AP in the interactive setting, establishing new state-of-the-art performance. The improvements are particularly pronounced in challenging scenarios: on LVIS-Ambiguous, a benchmark specifically designed to test disambiguation capabilities, the framework shows a remarkable +7.9 AP gain over its one-shot baseline after just a single corrective feedback iteration.

The efficiency metrics are equally impressive. Each interaction cycle adds less than 30ms of latency, making the system practical for real-time applications. This minimal overhead stems from the efficient memory update mechanism and the lightweight nature of the contrastive alignment computation.

Ablation studies reveal that both memory components are essential: removing positive anchors reduces performance by 4.2 AP, while eliminating negative constraints causes a 3.8 AP drop. The contrastive formulation outperforms alternative ranking strategies, including pure similarity-based and attention-based approaches.

Implications and Future Directions

IntRec represents a paradigm shift from static to interactive object detection. By incorporating user feedback into the retrieval loop, the framework addresses real-world scenarios where initial queries may be ambiguous or underspecified. The dual-memory architecture provides a principled way to accumulate knowledge across interactions, effectively learning user intent on-the-fly without requiring additional training data.

The minimal latency overhead suggests potential for deployment in interactive applications such as robotic manipulation, augmented reality interfaces, and assistive technologies. The framework’s modular design also allows integration with various backbone detectors, making it adaptable to future advances in open-vocabulary detection.

Future work could explore multi-modal feedback mechanisms (e.g., pointing gestures, verbal corrections), extend the approach to video sequences where temporal consistency matters, and investigate how the Intent State could be personalized to individual user preferences over longer interaction histories.

Takeaways

1. Interactive refinement through user feedback significantly outperforms one-shot detection, especially in ambiguous scenarios (+7.9 AP on LVIS-Ambiguous)
2. Dual-memory architecture (positive anchors + negative constraints) provides a principled framework for modeling user intent
3. Contrastive alignment enables fine-grained disambiguation by simultaneously maximizing similarity to confirmed objects and minimizing similarity to rejected ones
4. Real-time performance is maintained with <30ms latency per interaction, making the approach practical for interactive applications
5. The framework achieves state-of-the-art results on LVIS (35.4 AP) while remaining agnostic to the underlying detection backbone