VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

TL;DR

VDInstruct introduces a content-aware tokenization approach for zero-shot key information extraction from visual documents, achieving state-of-the-art accuracy while significantly reducing computational redundancy.

Contribution

It proposes a novel content-aware tokenization method and a three-stage training paradigm, improving efficiency and accuracy in document understanding tasks.

Findings

Achieves SOTA results on KIE benchmarks.

Reduces image tokens by approximately 3.6x.

Surpasses strong baselines in zero-shot evaluations.

Abstract

Key Information Extraction (KIE) underpins the understanding of visual documents (e.g., receipts and contracts) by extracting precise semantic content and accurately capturing spatial structure. Yet existing multimodal large language models (MLLMs) often perform poorly on dense documents and rely on vision tokenization approaches that scale with image size, leading to redundant computation and memory inefficiency. To address these challenges, we introduce VDInstruct, an MLLM that separates spatial region detection from semantic feature extraction. Central to our model is a content-aware tokenization strategy: rather than fragmenting the entire image uniformly, it generates tokens in proportion to document complexity, preserving critical structure while eliminating wasted tokens. Leveraging a three-stage training paradigm, our model achieves state-of-the-art (SOTA) results on KIE benchmarks, matching or exceeding the accuracy of leading approaches while reducing the number of image tokens by roughly 3.6x. In zero-shot evaluations, VDInstruct surpasses strong baselines-such as DocOwl 1.5-by +5.5 F1 points, highlighting its robustness to unseen documents. These findings show that content-aware tokenization combined with explicit layout modeling offers a promising direction forward for document understanding. Data, source code, and model weights will be made publicly available.