UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents

TL;DR

UNIKIE-BENCH provides a comprehensive benchmark for evaluating large multimodal models' ability to extract key information from diverse visual documents, revealing significant challenges and disparities in current model performance.

Contribution

It introduces a unified benchmark with two evaluation tracks for assessing LMMs on KIE tasks, highlighting persistent challenges and performance gaps.

Findings

Performance drops significantly with complex layouts and long-tail key fields.

Large disparities in model performance across document types and scenarios.

Grounding accuracy and layout reasoning remain challenging for LMMs.

Abstract

Key Information Extraction (KIE) from real-world documents remains challenging due to substantial variations in layout structures, visual quality, and task-specific information requirements. Recent Large Multimodal Models (LMMs) have shown promising potential for performing end-to-end KIE directly from document images. To enable a comprehensive and systematic evaluation across realistic and diverse application scenarios, we introduce UNIKIE-BENCH, a unified benchmark designed to rigorously evaluate the KIE capabilities of LMMs. UNIKIE-BENCH consists of two complementary tracks: a constrained-category KIE track with scenario-predefined schemas that reflect practical application needs, and an open-category KIE track that extracts any key information that is explicitly present in the document. Experiments on 15 state-of-the-art LMMs reveal substantial performance degradation under diverse schema definitions, long-tail key fields, and complex layouts, along with pronounced performance disparities across different document types and scenarios. These findings underscore persistent challenges in grounding accuracy and layout-aware reasoning for LMM-based KIE. All codes and datasets are available at https://github.com/NEUIR/UNIKIE-BENCH.