When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

TL;DR

This paper introduces an unsupervised self-evolution training framework for multimodal reasoning models that improves performance without relying on annotated data or external rewards, using a novel self-judgment mechanism.

Contribution

It presents a novel unsupervised training method leveraging self-judgment and trajectory sampling to enhance multimodal reasoning models without human annotations.

Findings

Consistent performance improvements on five mathematical reasoning benchmarks.

Effective use of self-consistency signals as training priors.

Robust policy updates via group-level relative scoring.

Abstract

Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.