Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning

TL;DR

This paper introduces a domain-specific framework for scientific table understanding that leverages dynamic image resolutions and specialized datasets, significantly improving numerical reasoning and generalization over existing models.

Contribution

It presents a comprehensive domain-specific dataset, instruction tuning, and a benchmark for scientific tables, enhancing multimodal models' capabilities in understanding and reasoning.

Findings

Superior performance with 52K scientific table images over 150K general tables

Significant improvements in numerical reasoning and generalization

Effective dynamic input resolution strategy enhances understanding

Abstract

Recent large language models (LLMs) have advanced table understanding capabilities but rely on converting tables into text sequences. While multimodal large language models (MLLMs) enable direct visual processing, they face limitations in handling scientific tables due to fixed input image resolutions and insufficient numerical reasoning capabilities. We present a comprehensive framework for multimodal scientific table understanding and reasoning with dynamic input image resolutions. Our framework consists of three key components: (1) MMSci-Pre, a domain-specific table structure learning dataset of 52K scientific table structure recognition samples, (2) MMSci-Ins, an instruction tuning dataset with 12K samples across three table-based tasks, and (3) MMSci-Eval, a benchmark with 3,114 testing samples specifically designed to evaluate numerical reasoning capabilities. Extensive experiments demonstrate that our domain-specific approach with 52K scientific table images achieves superior performance compared to 150K general-domain tables, highlighting the importance of data quality over quantity. Our proposed table-based MLLMs with dynamic input resolutions show significant improvements in both general table understanding and numerical reasoning capabilities, with strong generalisation to held-out datasets. Our code and data are publicly available at https://github.com/Bernard-Yang/MMSci_Table.