From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs

TL;DR

This paper introduces GRPO-Verif, a unified training approach for LLMs that jointly optimizes reasoning and self-verification, leading to improved self-verification ability without sacrificing reasoning performance.

Contribution

The paper proposes a novel joint optimization method for reasoning and self-verification in LLMs, enhancing their self-assessment capabilities.

Findings

Improved self-verification in LLMs

Maintained reasoning performance

Effective joint optimization approach

Abstract

The reasoning capabilities of large language models (LLMs) have been significantly improved through reinforcement learning (RL). Nevertheless, LLMs still struggle to consistently verify their own reasoning traces. This raises the research question of how to enhance the self-verification ability of LLMs and whether such an ability can further improve reasoning performance. In this work, we propose GRPO-Verif, an algorithm that jointly optimizes solution generation and self-verification within a unified loss function, with an adjustable hyperparameter controlling the weight of the verification signal. Experimental results demonstrate that our method enhances self-verification capability while maintaining comparable performance in reasoning.