Zero-Shot Document Understanding using Pseudo Table of Contents-Guided Retrieval-Augmented Generation

TL;DR

DocsRay is a training-free, multimodal document understanding system that uses pseudo-TOC generation and hierarchical retrieval to efficiently process complex documents with diverse elements, achieving high accuracy and reduced latency.

Contribution

Introduces DocsRay, a novel zero-shot document understanding framework combining pseudo-TOC generation with hierarchical retrieval, without requiring additional training or specialized models.

Findings

Reduced query latency by 45%

Achieved 64.7% accuracy on MMLongBench-Doc

Effectively processes multimodal documents with diverse elements

Abstract

Understanding complex multimodal documents remains challenging due to their structural inconsistencies and limited training data availability. We introduce \textit{DocsRay}, a training-free document understanding system that integrates pseudo Table of Contents (TOC) generation with hierarchical Retrieval-Augmented Generation (RAG). Our approach leverages multimodal Large Language Models' (LLMs) native capabilities to seamlessly process documents containing diverse elements such as text, images, charts, and tables without requiring specialized models or additional training. DocsRay's framework synergistically combines three key techniques: (1) a semantic structuring module using prompt-based LLM interactions to generate a hierarchical pseudo-TOC, (2) zero-shot multimodal analysis that converts diverse document elements into unified, text-centric representations using the inherent capabilities of multimodal LLMs, and (3) an efficient two-stage hierarchical retrieval system that reduces retrieval complexity from $O(N)$ to $O(S + k_1 \cdot N_s)$. Evaluated on documents averaging 49.4 pages and 20,971 textual tokens, DocsRay reduced query latency from 3.89 to 2.12 seconds, achieving a 45% efficiency improvement. On the MMLongBench-Doc benchmark, DocsRay-Pro attains an accuracy of 64.7%, substantially surpassing previous state-of-the-art results.