A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends

TL;DR

This survey reviews recent advances in Multimodal Large Language Models for Visually Rich Document Understanding, focusing on techniques, challenges, and future research directions.

Contribution

It provides a comprehensive overview of MLLM-based VRDU methods, highlighting emerging trends and identifying key challenges in the field.

Findings

MLLMs show promise in OCR-based and OCR-free document understanding.

Techniques for integrating textual, visual, and layout features are evolving.

Challenges include data scarcity and handling multi-page, multilingual documents.

Abstract

Visually Rich Document Understanding (VRDU) has become a pivotal area of research, driven by the need to automatically interpret documents that contain intricate visual, textual, and structural elements. Recently, Multimodal Large Language Models (MLLMs) have demonstrated significant promise in this domain, including both OCR-based and OCR-free approaches for information extraction from document images. This survey reviews recent advances in MLLM-based VRDU, highlighting emerging trends and promising research directions with a focus on two key aspects: (1) techniques for representing and integrating textual, visual, and layout features; (2) training paradigms, including pretraining, instruction tuning, and training strategies. Moreover, we address challenges such as data scarcity, handling multi-page and multilingual documents, and integrating emerging trends such as Retrieval-Augmented Generation and agentic frameworks. Our analysis offers a roadmap for advancing MLLM-based VRDU toward more scalable, reliable, and adaptable systems.