DRISHTIKON: Visual Grounding at Multiple Granularities in Documents

TL;DR

DRISHTIKON introduces a multi-granular visual grounding framework for complex, multilingual documents, improving interpretability and accuracy in document understanding and VQA tasks through innovative region matching and a new benchmark.

Contribution

The paper presents a novel multi-granular visual grounding approach combining OCR, language models, and region matching, along with a new benchmark dataset for evaluation.

Findings

Achieves state-of-the-art grounding accuracy.

Line-level granularity offers optimal balance of precision and recall.

Multi-block and multi-line reasoning improve performance.

Abstract

Visual grounding in text-rich document images is a critical yet underexplored challenge for Document Intelligence and Visual Question Answering (VQA) systems. We present DRISHTIKON, a multi-granular and multi-block visual grounding framework designed to enhance interpretability and trust in VQA for complex, multilingual documents. Our approach integrates multilingual OCR, large language models, and a novel region matching algorithm to localize answer spans at the block, line, word, and point levels. We introduce the Multi-Granular Visual Grounding (MGVG) benchmark, a curated test set of diverse circular notifications from various sectors, each manually annotated with fine-grained, human-verified labels across multiple granularities. Extensive experiments show that our method achieves state-of-the-art grounding accuracy, with line-level granularity providing the best balance between precision and recall. Ablation studies further highlight the benefits of multi-block and multi-line reasoning. Comparative evaluations reveal that leading vision-language models struggle with precise localization, underscoring the effectiveness of our structured, alignment-based approach. Our findings pave the way for more robust and interpretable document understanding systems in real-world, text-centric scenarios with multi-granular grounding support. Code and dataset are made available for future research.