NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

Paper: 2602.22144 Authors: Lingfeng Ren, Weihao Yu, Runpeng Yu, Xinchao Wang Categories: cs.CV, cs.AI, cs.CL

Abstract

Object hallucination remains a critical challenge in Large Vision-Language Models (LVLMs), where models generate descriptions of objects that are not present in the input image. This paper investigates the fundamental question of which component in the LVLM pipeline—the vision encoder or the language decoder—primarily contributes to these hallucinations. Through systematic experimentation, the authors demonstrate that object hallucinations are predominantly driven by strong language priors in the decoder rather than visual perception failures. Based on this insight, they propose NoLan (No-Language-Hallucination Decoding), a training-free framework that dynamically suppresses language priors by modulating the output distribution based on differences between multimodal and text-only inputs. Experimental results show substantial improvements across various LVLMs, with accuracy gains of up to 6.45 and 7.21 on the POPE benchmark for LLaVA-1.5 7B and Qwen-VL 7B respectively.

Key Contributions

• Systematic analysis identifying language decoder priors as the primary source of object hallucinations in LVLMs, rather than vision encoder limitations
• NoLan framework: a simple, training-free decoding strategy that dynamically suppresses language priors during generation
• Modulation mechanism based on output distribution differences between multimodal ($p_{\text{LVLM}}$) and text-only ($p_{\text{LLM}}$) inputs
• Comprehensive evaluation demonstrating consistent hallucination reduction across multiple LVLMs and benchmarks
• Open-source implementation enabling easy integration with existing models

Methodology

The NoLan framework operates during the decoding phase without requiring any model retraining or fine-tuning. The core mechanism involves computing a refined output distribution by comparing the LVLM’s multimodal predictions with the language model’s text-only predictions.

The refined probability distribution is computed as:

$p_{\text{NoLan}}(y_t | y_{<t}, x_v, x_t) = p_{\text{LVLM}}(y_t | y_{<t}, x_v, x_t) - \alpha \cdot \Delta p_t$

where $\Delta p_t = p_{\text{LVLM}}(y_t | y_{<t}, x_v, x_t) - p_{\text{LLM}}(y_t | y_{<t}, x_t)$ represents the distribution difference, and $\alpha$ is a dynamic modulation coefficient.

The key insight is that when the LVLM’s prediction strongly aligns with the text-only LLM prediction, it suggests the model is relying heavily on language priors rather than visual information. By suppressing these language-driven predictions, NoLan encourages the model to ground its outputs more firmly in the visual input.

The dynamic modulation coefficient $\alpha$ is adjusted based on the magnitude of $\Delta p_t$, allowing the framework to adaptively control the suppression strength. When the distribution difference is large (indicating strong visual grounding), suppression is minimal. When the difference is small (indicating language prior dominance), suppression increases.

Experimental Results

NoLan demonstrates consistent and substantial improvements across multiple benchmarks and model architectures:

POPE Benchmark: On the Polling-based Object Probing Evaluation, NoLan achieves accuracy improvements of 6.45 points for LLaVA-1.5 7B and 7.21 points for Qwen-VL 7B. These gains represent significant reductions in false positive object mentions.

Cross-Model Generalization: The framework shows effectiveness across diverse LVLM architectures including LLaVA variants, Qwen-VL, and InstructBLIP, indicating that the language prior issue is a common phenomenon rather than model-specific.

Task Versatility: Beyond object existence verification, NoLan improves performance on various vision-language tasks including image captioning, visual question answering, and detailed image description, demonstrating broad applicability.

Ablation Studies: The authors validate the importance of dynamic modulation versus fixed suppression coefficients, showing that adaptive adjustment based on distribution differences is crucial for optimal performance. Static suppression can over-correct and harm performance on samples where language priors are appropriate.

Implications and Future Directions

This work provides important insights into the mechanisms underlying object hallucinations in LVLMs. By identifying language priors as the primary culprit, it shifts focus from improving vision encoders to better balancing multimodal information during decoding.

The training-free nature of NoLan makes it immediately applicable to existing deployed models without requiring expensive retraining. This practical advantage could accelerate adoption in production systems where hallucination mitigation is critical.

However, several questions remain open. The optimal balance between language priors and visual grounding may vary across different task types and domains. Future work could explore task-adaptive modulation strategies or learn the suppression dynamics from data.

Additionally, while NoLan addresses object hallucinations, other types of hallucinations (attribute errors, relationship mistakes, contextual inconsistencies) may require complementary approaches. Investigating whether similar prior suppression strategies apply to these phenomena represents a promising research direction.

The framework also raises interesting questions about the role of language priors in multimodal learning more broadly. Are strong language priors inherently problematic, or do they serve important functions in certain contexts? Understanding this trade-off could inform future LVLM architecture designs.

Takeaways

1. Object hallucinations in LVLMs stem primarily from strong language decoder priors rather than vision encoder failures, challenging common assumptions about the source of these errors.

2. NoLan provides a training-free solution that can be immediately applied to existing models, making hallucination mitigation practical and accessible without expensive retraining.

3. Dynamic modulation of language prior suppression based on distribution differences between multimodal and text-only predictions is more effective than static suppression strategies.

4. The framework achieves substantial improvements across multiple models and benchmarks, with accuracy gains exceeding 6-7 points on POPE for popular LVLMs like LLaVA-1.5 and Qwen-VL.

5. This work opens new research directions in understanding and controlling the balance between language priors and visual grounding in multimodal models, with implications for future architecture design and training strategies.