Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning
Bohao Yang, Yingji Zhang, Dong Liu, Andr\'e Freitas, Chenghua Lin

TL;DR
This paper introduces a domain-specific framework for scientific table understanding that leverages dynamic image resolutions and specialized datasets, significantly improving numerical reasoning and generalization over existing models.
Contribution
It presents a comprehensive domain-specific dataset, instruction tuning, and a benchmark for scientific tables, enhancing multimodal models' capabilities in understanding and reasoning.
Findings
Superior performance with 52K scientific table images over 150K general tables
Significant improvements in numerical reasoning and generalization
Effective dynamic input resolution strategy enhances understanding
Abstract
Recent large language models (LLMs) have advanced table understanding capabilities but rely on converting tables into text sequences. While multimodal large language models (MLLMs) enable direct visual processing, they face limitations in handling scientific tables due to fixed input image resolutions and insufficient numerical reasoning capabilities. We present a comprehensive framework for multimodal scientific table understanding and reasoning with dynamic input image resolutions. Our framework consists of three key components: (1) MMSci-Pre, a domain-specific table structure learning dataset of 52K scientific table structure recognition samples, (2) MMSci-Ins, an instruction tuning dataset with 12K samples across three table-based tasks, and (3) MMSci-Eval, a benchmark with 3,114 testing samples specifically designed to evaluate numerical reasoning capabilities. Extensive…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Text Analysis Techniques · Semantic Web and Ontologies
