Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding

TL;DR

The paper introduces PinPoint, a two-stage framework that efficiently identifies and refines instruction-relevant image regions, enhancing reasoning accuracy while reducing computational costs in complex multimodal tasks.

Contribution

It proposes a novel instruction-region alignment method and new annotations for better localization of relevant regions in complex images, improving efficiency and accuracy.

Findings

Achieves superior accuracy over existing methods.

Reduces computational overhead significantly.

Provides richer supervision through new annotations.

Abstract

Large Vision-Language Models (LVLMs) have shown strong performance across various multimodal tasks by leveraging the reasoning capabilities of Large Language Models (LLMs). However, processing visually complex and information-rich images, such as infographics or document layouts, requires these models to generate a large number of visual tokens, leading to significant computational overhead. To address this, we propose PinPoint, a novel two-stage framework that first identifies instruction-relevant image regions and then refines them to extract fine-grained visual features for improved reasoning and efficiency. Central to our approach is the Instruction-Region Alignment, which localizes relevant regions using both visual input and textual instructions. We further introduce new annotations that provide richer ground-truth supervision for instruction-relevant regions across challenging VQA benchmarks: InfographicVQA, MultiPageDocVQA, and SinglePageDocVQA. Experimental results show that PinPoint not only achieves superior accuracy compared to existing methods but also reduces computational overhead by minimizing irrelevant visual tokens.