Index Light, Reason Deep: Deferred Visual Ingestion for Visual-Dense Document Question Answering

TL;DR

This paper introduces Deferred Visual Ingestion (DVI), a novel framework for visual-dense document question answering that improves accuracy by deferring visual analysis to inference time and leveraging document structure for indexing.

Contribution

The paper proposes DVI, which avoids VLM calls during preprocessing, uses hierarchical indexing based on document structure, and targets visual analysis during inference, significantly enhancing QA performance on engineering documents.

Findings

DVI achieves 65.6% accuracy on Bridge engineering drawings, outperforming PI's 24.3%.

On Steel catalog, DVI reaches 30.6% accuracy, surpassing 16.1%.

On CircuitVQA, DVI's retrieval ImgR@3 is 31.2%, much higher than 0.7%.

Abstract

Existing multimodal document question answering methods predominantly adopt a Pre-Ingestion (PI) strategy: during the indexing phase, a Vision Language Model (VLM) is called on every page to generate page descriptions that are then encoded into vectors, and questions are answered via embedding similarity retrieval. However, this approach faces a dual dilemma on visual-dense engineering documents: VLM blind descriptions inevitably lose critical visual details, and embedding retrieval systematically fails on highly similar documents. This paper proposes the Deferred Visual Ingestion (DVI) framework: zero VLM calls during preprocessing, leveraging only document structural information (table of contents, drawing numbers) to automatically build a hierarchical index through the HDNC (Hierarchical Drawing Number Clustering) algorithm; during inference, candidate pages are located via BM25 retrieval, and the original images along with the specific question are sent to a VLM for targeted analysis. Large-scale experiments on three datasets validate the effectiveness of DVI: on Bridge engineering drawings (1,323 questions), end-to-end QA accuracy reaches 65.6\% vs. PI's 24.3\% (+41.3pp); on Steel catalog (186 questions), 30.6\% vs. 16.1\% (+14.5pp); on CircuitVQA, a public benchmark (9,315 questions), retrieval ImgR@3 achieves 31.2\% vs. 0.7\%. On the Bridge dataset, we evaluated ColPali (ICLR 2025 visual retrieval SOTA), which achieved only 20.1\% PageR@3, demonstrating that the failure of embedding retrieval on homogeneous engineering documents is structural rather than due to insufficient model capability. Ablation studies show that HDNC zero-cost automatic indexing yields a +27.5pp retrieval improvement, and VLM conversion rate analysis confirms that the bottleneck lies on the retrieval side rather than the comprehension side.