VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integration

TL;DR

This paper introduces VisTIRA, a structured reasoning framework that combines tool integration and OCR grounding to significantly improve visual math reasoning in vision-language models, reducing the modality gap with text-based models.

Contribution

The paper presents VisTIRA, a novel framework that decomposes image-based math problems into natural language and executable steps, along with a LaTeX pipeline and synthetic data for training and evaluation.

Findings

Tool-integrated supervision enhances reasoning accuracy.

OCR grounding benefits smaller models more significantly.

The modality gap decreases as model size increases.

Abstract

Vision-language models (VLMs) lag behind text-only language models on mathematical reasoning when the same problems are presented as images rather than text. We empirically characterize this as a modality gap: the same question in text form yields markedly higher accuracy than its visually typeset counterpart, due to compounded failures in reading dense formulas, layout, and mixed symbolic-diagrammatic context. First, we introduce VisTIRA (Vision and Tool-Integrated Reasoning Agent), a tool-integrated reasoning framework that enables structured problem solving by iteratively decomposing a given math problem (as an image) into natural language rationales and executable Python steps to determine the final answer. Second, we build a framework to measure and improve visual math reasoning: a LaTeX-based pipeline that converts chain-of-thought math corpora (e.g., NuminaMath) into challenging image counterparts, and a large set of synthetic tool-use trajectories derived from a real-world, homework-style image dataset (called SnapAsk) for fine-tuning VLMs. Our experiments show that tool-integrated supervision improves image-based reasoning, and OCR grounding can further narrow the gap for smaller models, although its benefit diminishes at scale. These findings highlight that modality gap severity inversely correlates with model size, and that structured reasoning and OCR-based grounding are complementary strategies for advancing visual mathematical reasoning.