Integrating Visual Interpretation and Linguistic Reasoning for Math Problem Solving

TL;DR

This paper proposes a decoupled reasoning framework combining visual interpretation models and large language models to improve multi-modal reasoning, especially in complex math problems, offering a cost-effective and flexible alternative to end-to-end LVLMs.

Contribution

It introduces a novel decoupled reasoning paradigm that leverages existing visual and linguistic models, enhancing performance on vision-language tasks without extensive end-to-end training.

Findings

Outperforms recent LVLMs on benchmarks

Achieves significant gains on geometric math problems

Demonstrates cost-efficient multi-modal reasoning

Abstract

Current large vision-language models (LVLMs) typically employ a connector module to link visual features with text embeddings of large language models (LLMs) and use end-to-end training to achieve multi-modal understanding in a unified process. Effective alignment needs high-quality pre-training data and a carefully designed training process. Current LVLMs face challenges when addressing complex vision-language reasoning tasks, with their reasoning capabilities notably lagging behind those of LLMs. This paper proposes a paradigm shift: instead of training end-to-end vision-language reasoning models, we advocate for developing a decoupled reasoning framework based on existing visual interpretation specialists and text-based reasoning LLMs. Our approach leverages (1) a dedicated vision-language model to transform the visual content of images into textual descriptions and (2) an LLM to perform reasoning according to the visual-derived text and the original question. This method presents a cost-efficient solution for multi-modal model development by optimizing existing models to work collaboratively, avoiding end-to-end development of vision-language models from scratch. By transforming images into language model-compatible text representations, it facilitates future low-cost and flexible upgrades to upcoming powerful LLMs. We introduce an outcome-rewarded joint-tuning strategy to optimize the cooperation between the visual interpretation and linguistic reasoning model. Evaluation results on vision-language benchmarks demonstrate that the decoupled reasoning framework outperforms recent LVLMs. Our approach yields particularly significant performance gains on visually intensive geometric mathematics problems. The code is available: https://github.com/guozix/DVLR.