Efficient Table Retrieval and Understanding with Multimodal Large Language Models

TL;DR

This paper introduces TabRAG, a framework that combines visual-text retrieval and large language models to identify and reason over relevant table images from large collections, significantly improving accuracy in real-world table understanding tasks.

Contribution

The paper presents a novel retrieval and reasoning framework for large collections of table images using multimodal models, addressing practical challenges in table understanding.

Findings

Outperforms existing methods by 7.0% in retrieval recall.

Achieves 6.1% higher answer accuracy.

Demonstrates effectiveness on a large-scale dataset with 88,161 training samples.

Abstract

Tabular data is frequently captured in image form across a wide range of real-world scenarios such as financial reports, handwritten records, and document scans. These visual representations pose unique challenges for machine understanding, as they combine both structural and visual complexities. While recent advances in Multimodal Large Language Models (MLLMs) show promising results in table understanding, they typically assume the relevant table is readily available. However, a more practical scenario involves identifying and reasoning over relevant tables from large-scale collections to answer user queries. To address this gap, we propose TabRAG, a framework that enables MLLMs to answer queries over large collections of table images. Our approach first retrieves candidate tables using jointly trained visual-text foundation models, then leverages MLLMs to perform fine-grained reranking of these candidates, and finally employs MLLMs to reason over the selected tables for answer generation. Through extensive experiments on a newly constructed dataset comprising 88,161 training and 9,819 testing samples across 8 benchmarks with 48,504 unique tables, we demonstrate that our framework significantly outperforms existing methods by 7.0% in retrieval recall and 6.1% in answer accuracy, offering a practical solution for real-world table understanding tasks.