DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming

TL;DR

DocKylin is a multimodal model designed for visual document understanding that employs adaptive pixel and token slimming techniques to improve efficiency and performance on complex document images.

Contribution

The paper introduces DocKylin, a novel multimodal model with adaptive pixel and token slimming modules for efficient visual document understanding.

Findings

Effective visual content slimming reduces computational costs.

Improved performance on VDU benchmarks.

Adaptive modules enhance detail perception and efficiency.

Abstract

Current multimodal large language models (MLLMs) face significant challenges in visual document understanding (VDU) tasks due to the high resolution, dense text, and complex layouts typical of document images. These characteristics demand a high level of detail perception ability from MLLMs. While increasing input resolution improves detail perception capability, it also leads to longer sequences of visual tokens, increasing computational costs and straining the models' ability to handle long contexts. To address these challenges, we introduce DocKylin, a document-centric MLLM that performs visual content slimming at both the pixel and token levels, thereby reducing token sequence length in VDU scenarios. We introduce an Adaptive Pixel Slimming (APS) preprocessing module to perform pixel-level slimming, increasing the proportion of informative pixels. Moreover, we propose a novel Dynamic Token Slimming (DTS) module to conduct token-level slimming, filtering essential tokens and removing others to adaptively create a more compact visual sequence. Experiments demonstrate DocKylin's promising performance across various VDU benchmarks and the effectiveness of each component.