Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation

TL;DR

This paper introduces Chain of Evidence (CoE), a visual attribution framework for iRAG that reasons directly over document images, providing pixel-level evidence visualization and outperforming text-based methods on visual-rich datasets.

Contribution

CoE is a retriever-agnostic visual attribution method that leverages Vision-Language Models to directly analyze document images, eliminating parsing bottlenecks and enhancing interpretability.

Findings

CoE outperforms text-based baselines on visual-rich datasets.

Fine-tuned Qwen3-VL-8B-Instruct achieves robust performance.

CoE provides pixel-level evidence visualization for reasoning.

Abstract

Iterative Retrieval-Augmented Generation (iRAG) has emerged as a powerful paradigm for answering complex multi-hop questions by progressively retrieving and reasoning over external documents. However, current systems predominantly operate on parsed text, which creates two critical bottlenecks: (1) \textit{Coarse-grained attribution}, where users are burdened with manually locating evidence within lengthy documents based on vague text-level citations; and (2) \textit{Visual semantic loss}, where the conversion of visually rich documents (e.g., slides, PDFs with charts) into text discards spatial logic and layout cues essential for reasoning. To bridge this gap, we present \textbf{Chain of Evidence (CoE)}, a retriever-agnostic visual attribution framework that leverages Vision-Language Models to reason directly over screenshots of retrieved document candidates. CoE eliminates format-specific parsing and outputs precise bounding boxes, visualizing the complete reasoning chain within the retrieved candidate set. We evaluate CoE on two distinct benchmarks: \textbf{Wiki-CoE}, a large-scale dataset of structured web pages derived from 2WikiMultiHopQA, and \textbf{SlideVQA}, a challenging dataset of presentation slides featuring complex diagrams and free-form layouts. Experiments demonstrate that fine-tuned Qwen3-VL-8B-Instruct achieves robust performance, significantly outperforming text-based baselines in scenarios requiring visual layout understanding, while establishing a retriever-agnostic solution for pixel-level interpretable iRAG. Our code is available at https://github.com/PeiYangLiu/CoE.git.