ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction

TL;DR

ShredBench is a new benchmark for evaluating multimodal large language models' ability to reconstruct shredded documents, revealing significant challenges in semantic reasoning across visual discontinuities.

Contribution

The paper introduces ShredBench, a systematic evaluation framework with an automated pipeline for assessing VRDU capabilities of MLLMs on shredded documents across multiple languages and formats.

Findings

MLLMs perform well on intact documents but struggle with shredded content.

Performance drops sharply as fragmentation increases, indicating difficulty in visual-semantic reasoning.

Current MLLMs lack the fine-grained reasoning needed for robust document reconstruction.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable performance in Visually Rich Document Understanding (VRDU) tasks, but their capabilities are mainly evaluated on pristine, well-structured document images. We consider content restoration from shredded fragments, a challenging VRDU setting that requires integrating visual pattern recognition with semantic reasoning under significant content discontinuities. To facilitate systematic evaluation of complex VRDU tasks, we introduce ShredBench, a benchmark supported by an automated generation pipeline that renders fragmented documents directly from Markdown. The proposed pipeline ensures evaluation validity by allowing the flexible integration of latest or unseen textual sources to prevent training data contamination. ShredBench assesses four scenarios (English, Chinese, Code, Table) with three fragmentation granularities (8, 12, 16 pieces). Empirical evaluations on state-of-the-art MLLMs reveal a significant performance gap: The method is effective on intact documents; however, once the document is shredded, restoration becomes a significant challenge, with NED dropping sharply as fragmentation increases. Our findings highlight that current MLLMs lack the fine-grained cross-modal reasoning required to bridge visual discontinuities, identifying a critical gap in robust VRDU research.