A Multi-Granularity Retrieval Framework for Visually-Rich Documents

TL;DR

This paper introduces a multi-granularity multimodal retrieval framework for visually-rich documents, combining hierarchical encoding, modality-aware retrieval, and vision-language models to improve accuracy without fine-tuning.

Contribution

It presents a unified retrieval framework that effectively handles complex visual and textual data in documents, using off-the-shelf models and hybrid strategies for robust performance.

Findings

Achieves a top score of 65.56 in retrieval accuracy.

Enhances retrieval with layout-aware search and VLM-based verification.

Operates effectively without task-specific fine-tuning.

Abstract

Retrieval-augmented generation (RAG) systems have predominantly focused on text-based retrieval, limiting their effectiveness in handling visually-rich documents that encompass text, images, tables, and charts. To bridge this gap, we propose a unified multi-granularity multimodal retrieval framework tailored for two benchmark tasks: MMDocIR and M2KR. Our approach integrates hierarchical encoding strategies, modality-aware retrieval mechanisms, and vision-language model (VLM)-based candidate filtering to effectively capture and utilize the complex interdependencies between textual and visual modalities. By leveraging off-the-shelf vision-language models and implementing a training-free hybrid retrieval strategy, our framework demonstrates robust performance without the need for task-specific fine-tuning. Experimental evaluations reveal that incorporating layout-aware search and VLM-based candidate verification significantly enhances retrieval accuracy, achieving a top performance score of 65.56. This work underscores the potential of scalable and reproducible solutions in advancing multimodal document retrieval systems.