RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations

TL;DR

RobustVisRAG is a causality-guided framework that enhances vision-based retrieval-augmented generation models' robustness to visual degradations by separating semantics from distortions, validated on a new comprehensive benchmark.

Contribution

The paper introduces RobustVisRAG, a novel causality-aware dual-path approach with new training objectives and a large-scale degraded dataset, significantly improving robustness in visual retrieval and generation tasks.

Findings

Improves retrieval, generation, and end-to-end performance under visual degradations.

Maintains accuracy on clean inputs.

Introduces the Distortion-VisRAG benchmark with diverse real-world distortions.

Abstract

Vision-based Retrieval-Augmented Generation (VisRAG) leverages vision-language models (VLMs) to jointly retrieve relevant visual documents and generate grounded answers based on multimodal evidence. However, existing VisRAG models degrade in performance when visual inputs suffer from distortions such as blur, noise, low light, or shadow, where semantic and degradation factors become entangled within pretrained visual encoders, leading to errors in both retrieval and generation stages. To address this limitation, we introduce RobustVisRAG, a causality-guided dual-path framework that improves VisRAG robustness while preserving efficiency and zero-shot generalization. RobustVisRAG uses a non-causal path to capture degradation signals through unidirectional attention and a causal path to learn purified semantics guided by these signals. Together with the proposed Non-Causal Distortion Modeling and Causal Semantic Alignment objectives, the framework enforces a clear separation between semantics and degradations, enabling stable retrieval and generation under challenging visual conditions. To evaluate robustness under realistic conditions, we introduce the Distortion-VisRAG dataset, a large-scale benchmark containing both synthetic and real-world degraded documents across seven domains, with 12 synthetic and 5 real distortion types that comprehensively reflect practical visual degradations. Experimental results show that RobustVisRAG improves retrieval, generation, and end-to-end performance by 7.35%, 6.35%, and 12.40%, respectively, on real-world degradations, while maintaining comparable accuracy on clean inputs.