DOGR: Towards Versatile Visual Document Grounding and Referring

TL;DR

This paper introduces DOGR, a new dataset and benchmark for fine-grained visual document grounding and referring, leveraging a data engine to enhance multimodal language models' capabilities in detailed document understanding.

Contribution

It presents the DOGR-Engine for generating high-quality data, constructs the DOGR-Bench for comprehensive evaluation, and develops the DOGR baseline model to improve document grounding and referring tasks.

Findings

DOGR-Engine effectively generates detailed parsing and instruction data.

DOGR-Bench covers seven diverse grounding and referring tasks.

The DOGR model outperforms existing methods in fine-grained document understanding.

Abstract

With recent advances in Multimodal Large Language Models (MLLMs), grounding and referring capabilities have gained increasing attention for achieving detailed understanding and flexible user interaction. However, these capabilities still remain underdeveloped in visual document understanding due to the scarcity of fine-grained datasets and comprehensive benchmarks. To fill this gap, we propose the DOcument Grounding and Referring data engine (DOGR-Engine), which generates two types of high-quality fine-grained document data: (1) multi-granular parsing data to improve text localization and recognition, and (2) instruction-tuning data to activate MLLMs' grounding and referring capabilities in dialogue and reasoning. Using the DOGR-Engine, we construct DOGR-Bench, a benchmark covering seven grounding and referring tasks across three document types (chart, poster, and PDF document), offering a comprehensive evaluation of fine-grained document understanding. Leveraging the generated data, we further develop DOGR, a strong baseline model that excels in text localization and recognition, while precisely grounds and refers to key textual information during conversation and reasoning, thereby advancing document understanding to a finer granularity and enable flexible interaction paradigms.