RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations

Paper: 2602.22013 Authors: I-Hsiang Chen, Yu-Wei Liu, Tse-Yu Wu, Yu-Chien Chiang, Jen-Chien Yang, Wei-Ting Chen Categories: cs.CV

Abstract

Vision-based Retrieval-Augmented Generation (VisRAG) systems combine vision-language models to retrieve relevant visual documents and generate answers from multimodal evidence. However, these systems suffer significant performance degradation when visual inputs are corrupted by real-world distortions such as blur, noise, low lighting, or shadows. The core problem lies in the entanglement of semantic content and degradation artifacts within pretrained visual encoders, causing errors to propagate through both retrieval and generation stages. RobustVisRAG addresses this challenge through a causality-guided dual-path architecture that explicitly separates degradation signals from semantic information, enabling robust performance under challenging visual conditions while preserving zero-shot generalization capabilities.

Key Contributions

• Causality-Guided Dual-Path Architecture: Introduces a novel framework with a non-causal path for capturing degradation signals via unidirectional attention and a causal path for learning purified semantics guided by these signals
• Disentanglement Objectives: Proposes Non-Causal Distortion Modeling (NCDM) and Causal Semantic Alignment (CSA) objectives that enforce clear separation between semantic content and visual degradations
• Distortion-VisRAG Benchmark: Creates a comprehensive large-scale dataset spanning seven domains with 12 synthetic and 5 real-world distortion types, providing realistic evaluation conditions for robust VisRAG research
• Significant Performance Gains: Achieves 7.35% improvement in retrieval, 6.35% in generation, and 12.40% in end-to-end performance on real-world degradations while maintaining comparable accuracy on clean inputs

Technical Methodology

The RobustVisRAG framework operates through a sophisticated dual-path mechanism designed to disentangle visual degradations from semantic content. The architecture consists of two parallel processing streams that work in concert to achieve robust multimodal understanding.

The non-causal path employs unidirectional attention mechanisms to capture degradation-specific signals without allowing these artifacts to contaminate semantic representations. This path processes visual features through layers that explicitly model distortion characteristics, learning to identify and isolate corruption patterns such as blur kernels, noise distributions, and lighting variations. The Non-Causal Distortion Modeling objective trains this path to reconstruct degradation parameters, ensuring that the model develops a comprehensive understanding of how visual quality affects the input.

The causal path focuses on extracting clean semantic representations by leveraging the degradation signals identified by the non-causal path. Through bidirectional attention guided by distortion awareness, this path learns to filter out corruption artifacts and maintain semantic consistency. The Causal Semantic Alignment objective ensures that semantic features remain stable across different degradation levels, enforcing invariance to visual quality while preserving content-specific information necessary for accurate retrieval and generation.

The framework integrates seamlessly with existing vision-language models, requiring no architectural changes to the base VLM. During inference, both paths process the input simultaneously, with the non-causal path providing degradation context that guides the causal path’s semantic extraction. This design maintains computational efficiency while significantly improving robustness, as the additional processing overhead is minimal compared to the base model’s computation.

Experimental Results and Analysis

The evaluation on the Distortion-VisRAG benchmark reveals substantial improvements across multiple metrics and degradation types. On real-world distortions, RobustVisRAG demonstrates remarkable resilience, with retrieval accuracy improving by 7.35% over baseline VisRAG models. This improvement is particularly pronounced in scenarios involving multiple simultaneous degradations, such as documents affected by both low lighting and motion blur, where traditional approaches experience catastrophic performance drops.

Generation quality metrics show a 6.35% improvement, measured through both automated metrics and human evaluation. The model maintains factual accuracy and coherence even when source documents are severely degraded, indicating that the semantic disentanglement successfully preserves content understanding despite visual corruption. The end-to-end pipeline performance, combining retrieval and generation, achieves a 12.40% improvement, demonstrating that benefits compound across the full VisRAG workflow.

Critically, these robustness gains come without sacrificing performance on clean inputs. The model maintains comparable accuracy to baseline systems on undistorted documents, validating that the causality-guided approach does not introduce harmful biases or overfitting to degraded data. Ablation studies confirm that both the non-causal and causal paths contribute essential functionality, with removal of either component resulting in significant performance degradation.

The benchmark evaluation across synthetic and real-world distortions reveals interesting patterns in model behavior. Synthetic distortions with well-defined mathematical models (Gaussian blur, additive noise) show strong performance improvements, while real-world degradations (camera shake, compression artifacts, environmental lighting) demonstrate even larger gains, suggesting that the model learns generalizable robustness principles rather than overfitting to specific corruption types.

Implications and Future Directions

RobustVisRAG addresses a critical gap in practical deployment of vision-based RAG systems. Real-world document collections frequently contain images captured under suboptimal conditions—scanned documents with artifacts, photographs taken in poor lighting, compressed images with quality loss, or historical documents with age-related degradation. By enabling robust performance under these conditions, the framework significantly expands the applicability of VisRAG to real-world scenarios where perfect visual quality cannot be guaranteed.

The causality-guided disentanglement approach offers broader implications for multimodal learning. The principle of explicitly separating task-relevant semantics from nuisance factors through architectural design and targeted objectives could extend to other domains where input quality varies, such as audio processing with background noise, text understanding with OCR errors, or sensor data with measurement uncertainty.

Future research directions include extending the framework to handle temporal degradations in video-based RAG, incorporating adaptive mechanisms that adjust processing based on detected degradation severity, and exploring whether similar disentanglement principles can improve robustness to adversarial perturbations. The Distortion-VisRAG benchmark provides a foundation for continued research in robust multimodal understanding, enabling systematic evaluation of future approaches.

Takeaways

1. Visual degradations significantly impair VisRAG performance by entangling semantic content with corruption artifacts in pretrained encoders, causing errors in both retrieval and generation stages
2. Causality-guided dual-path architectures can effectively disentangle degradation signals from semantic information through specialized attention mechanisms and targeted training objectives
3. Explicit modeling of distortions via non-causal paths enables causal paths to learn purified semantic representations that remain stable across varying visual quality levels
4. Real-world degradations pose greater challenges than synthetic distortions, but causality-aware approaches show even stronger improvements on realistic corruption patterns
5. Robustness to visual degradations can be achieved without sacrificing performance on clean inputs, enabling practical deployment in mixed-quality document collections
6. The Distortion-VisRAG benchmark provides comprehensive evaluation across 17 distortion types spanning synthetic and real-world scenarios, establishing a standard for robust VisRAG research
7. Performance improvements compound across the full VisRAG pipeline, with end-to-end gains (12.40%) exceeding individual component improvements, demonstrating system-level benefits of robustness