Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMs

TL;DR

This paper identifies the critical role of fine-grained visual understanding in mathematical reasoning within multimodal large language models and introduces a novel vision encoder to improve geometric primitive recognition.

Contribution

The paper proposes SVE-Math, a new model with a geometric-grounded vision encoder and feature router, enhancing visual primitive recognition and reasoning in MLLMs.

Findings

SVE-Math outperforms other 7B models by 15% on MathVerse.

SVE-Math achieves competitive results on GeoQA despite smaller training datasets.

Advanced models like GPT-4o have a 70% error rate in geometric entity recognition.

Abstract

Current multimodal large language models (MLLMs) often underperform on mathematical problem-solving tasks that require fine-grained visual understanding. The limitation is largely attributable to inadequate perception of geometric primitives during image-level contrastive pre-training (e.g., CLIP). While recent efforts to improve math MLLMs have focused on scaling up mathematical visual instruction datasets and employing stronger LLM backbones, they often overlook persistent errors in visual recognition. In this paper, we systematically evaluate the visual grounding capabilities of state-of-the-art MLLMs and reveal a significant negative correlation between visual grounding accuracy and problem-solving performance, underscoring the critical role of fine-grained visual understanding. Notably, advanced models like GPT-4o exhibit a 70% error rate when identifying geometric entities, highlighting that this remains a key bottleneck in visual mathematical reasoning. To address this, we propose a novel approach, SVE-Math (Selective Vision-Enhanced Mathematical MLLM), featuring a geometric-grounded vision encoder and a feature router that dynamically adjusts the contribution of hierarchical visual feature maps. Our model recognizes accurate visual primitives and generates precise visual prompts tailored to the language model's reasoning needs. In experiments, SVE-Math-Qwen2.5-7B outperforms other 7B models by 15% on MathVerse and is compatible with GPT-4V on MathVista. Despite being trained on smaller datasets, SVE-Math-7B achieves competitive performance on GeoQA, rivaling models trained on significantly larger datasets. Our findings emphasize the importance of incorporating fine-grained visual understanding into MLLMs and provide a promising direction for future research.