DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding

TL;DR

DocLayLLM is an efficient multi-modal extension of large language models designed for text-rich document understanding, integrating visual and positional information to improve performance with lightweight training.

Contribution

It introduces a novel lightweight multi-modal extension of LLMs with chain-of-thought techniques for improved document understanding.

Findings

Outperforms existing OCR-dependent methods.

Achieves high performance with lightweight training.

Effectively integrates visual and positional tokens.

Abstract

Text-rich document understanding (TDU) requires comprehensive analysis of documents containing substantial textual content and complex layouts. While Multimodal Large Language Models (MLLMs) have achieved fast progress in this domain, existing approaches either demand significant computational resources or struggle with effective multi-modal integration. In this paper, we introduce DocLayLLM, an efficient multi-modal extension of LLMs specifically designed for TDU. By lightly integrating visual patch tokens and 2D positional tokens into LLMs' input and encoding the document content using the LLMs themselves, we fully take advantage of the document comprehension capability of LLMs and enhance their perception of OCR information. We have also deeply considered the role of chain-of-thought (CoT) and innovatively proposed the techniques of CoT Pre-training and CoT Annealing. Our DocLayLLM can achieve remarkable performances with lightweight training settings, showcasing its efficiency and effectiveness. Experimental results demonstrate that our DocLayLLM outperforms existing OCR-dependent methods and OCR-free competitors. Code and model are available at https://github.com/whlscut/DocLayLLM.