Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs

TL;DR

This paper introduces TGDoc, a model that enhances multimodal large language models with text-grounding capabilities, significantly improving understanding of text-rich images by integrating spatial text location information.

Contribution

The paper presents a novel text-grounding approach for document understanding, including a new dataset, instruction tuning tasks, and state-of-the-art performance on benchmarks.

Findings

Text-grounding improves model interpretation of text-rich images.

TGDoc achieves state-of-the-art results on multiple benchmarks.

The dataset of 99K PowerPoint presentations supports training and evaluation.

Abstract

In the field of document understanding, significant advances have been made in the fine-tuning of Multimodal Large Language Models (MLLMs) with instruction-following data. Nevertheless, the potential of text-grounding capability within text-rich scenarios remains underexplored. In this paper, we present a text-grounding document understanding model, termed TGDoc, which addresses this deficiency by enhancing MLLMs with the ability to discern the spatial positioning of text within images. Empirical evidence suggests that text-grounding improves the model's interpretation of textual content, thereby elevating its proficiency in comprehending text-rich images. Specifically, we compile a dataset containing 99K PowerPoint presentations sourced from the internet. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and large language model. Moreover, we curate a collection of text-rich images and prompt the text-only GPT-4 to generate 12K high-quality conversations, featuring textual locations within text-rich scenarios. By integrating text location data into the instructions, TGDoc is adept at discerning text locations during the visual question process. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.