Docopilot: Improving Multimodal Models for Document-Level Understanding

TL;DR

This paper introduces Docopilot, a native multimodal model trained on a new large-scale dataset, Doc-750K, that significantly improves document-level understanding without relying on retrieval-augmented methods.

Contribution

The paper presents a high-quality, diverse dataset for document comprehension and a native multimodal model that effectively captures cross-page dependencies.

Findings

Docopilot outperforms existing models in coherence and accuracy.

The dataset enables better understanding of complex, multi-page documents.

The model achieves efficient, multi-turn document interactions.

Abstract

Despite significant progress in multimodal large language models (MLLMs), their performance on complex, multi-page document comprehension remains inadequate, largely due to the lack of high-quality, document-level datasets. While current retrieval-augmented generation (RAG) methods offer partial solutions, they suffer from issues, such as fragmented retrieval contexts, multi-stage error accumulation, and extra time costs of retrieval. In this work, we present a high-quality document-level dataset, Doc-750K, designed to support in-depth understanding of multimodal documents. This dataset includes diverse document structures, extensive cross-page dependencies, and real question-answer pairs derived from the original documents. Building on the dataset, we develop a native multimodal model, Docopilot, which can accurately handle document-level dependencies without relying on RAG. Experiments demonstrate that Docopilot achieves superior coherence, accuracy, and efficiency in document understanding tasks and multi-turn interactions, setting a new baseline for document-level multimodal understanding. Data, code, and models are released at https://github.com/OpenGVLab/Docopilot