Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

TL;DR

This paper introduces a preference optimization method to improve multimodal reasoning in large language models, significantly enhancing their Chain-of-Thought performance and achieving state-of-the-art results on reasoning benchmarks.

Contribution

The paper proposes Mixed Preference Optimization (MPO) and a large-scale multimodal reasoning dataset, improving reasoning capabilities of MLLMs beyond existing fine-tuning methods.

Findings

InternVL2-8B-MPO achieves 67.0 accuracy on MathVista.

MPO boosts multimodal Chain-of-Thought performance.

Model performance approaches that of much larger models.

Abstract

Existing open-source multimodal large language models (MLLMs) generally follow a training process involving pre-training and supervised fine-tuning. However, these models suffer from distribution shifts, which limit their multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance. To address this, we introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data side, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset; and (2) on the model side, we explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance. Our approach enhances the multimodal reasoning abilities of both InternVL2-8B and InternVL2-76B. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance comparable to the 10$\times$ larger InternVL2-76B. We hope this study could inspire further advancements in MLLMs. Code, data, and model are released.