Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

TL;DR

This paper introduces Thinking with Tables (TWT), a neuro-symbolic reasoning approach that significantly improves multimodal understanding of tabular data by addressing structural variability, feature dependencies, and heterogeneity in downstream tasks.

Contribution

The paper presents a novel program-aided neuro-symbolic reasoning framework for enhanced multi-modal tabular understanding, outperforming existing methods on multiple datasets.

Findings

TWT outperforms baselines by 10% in accuracy.

TWT matches or surpasses proprietary SOTA LLMs on TVMU tasks.

Effective handling of structural variability and feature dependencies.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10\% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks. Models and codes are available at https://github.com/kunyang-YU/Thinking-with-Tables