Describe-then-Reason: Improving Multimodal Mathematical Reasoning through Visual Comprehension Training

TL;DR

This paper introduces VCAR, a two-step training pipeline that enhances multimodal large language models' visual comprehension and mathematical reasoning abilities by combining visual description generation with rationale training, leading to improved performance on visual math tasks.

Contribution

The paper proposes VCAR, a novel two-step training approach that emphasizes visual comprehension to improve multimodal mathematical reasoning in large language models.

Findings

VCAR significantly outperforms baseline methods on visual math benchmarks.

Visual description generation improves models' understanding of math figures.

Enhanced visual training leads to better reasoning accuracy in complex problems.

Abstract

Open-source multimodal large language models (MLLMs) excel in various tasks involving textual and visual inputs but still struggle with complex multimodal mathematical reasoning, lagging behind proprietary models like GPT-4V(ision) and Gemini-Pro. Although fine-tuning with intermediate steps (i.e., rationales) elicits some mathematical reasoning skills, the resulting models still fall short in visual comprehension due to inadequate visual-centric supervision, which leads to inaccurate interpretation of math figures. To address this issue, we propose a two-step training pipeline VCAR, which emphasizes the Visual Comprehension training in Addition to mathematical Reasoning learning. It first improves the visual comprehension ability of MLLMs through the visual description generation task, followed by another training step on generating rationales with the assistance of descriptions. Experimental results on two popular benchmarks demonstrate that VCAR substantially outperforms baseline methods solely relying on rationale supervision, especially on problems with high visual demands.