Learning to Self-Verify Makes Language Models Better Reasoners

TL;DR

This paper investigates the asymmetry between generation and self-verification in large language models, revealing that training models to self-verify can enhance their reasoning and generation abilities.

Contribution

It introduces a multi-task reinforcement learning framework that jointly optimizes generation and self-verification, improving reasoning performance.

Findings

Self-verification training improves generation accuracy.

Models with integrated self-verification outperform generation-only models.

Self-verification enhances reasoning trace efficiency.

Abstract

Recent large language models (LLMs) achieve strong performance in generating promising reasoning paths for complex tasks. However, despite powerful generation ability, LLMs remain weak at verifying their own answers, revealing a persistent capability asymmetry between generation and self-verification. In this work, we conduct an in-depth investigation of this asymmetry throughout training evolution and show that, even on the same task, improving generation does not lead to corresponding improvements in self-verification. Interestingly, we find that the reverse direction of this asymmetry behaves differently: learning to self-verify can effectively improve generation performance, achieving accuracy comparable to standard generation training while yielding more efficient and effective reasoning traces. Building on this observation, we further explore integrating self-verification into generation training by formulating a multi-task reinforcement learning framework, where generation and self-verification are optimized as two independent but complementary objectives. Extensive experiments across benchmarks and models demonstrate performance gains over generation-only training in both generation and verification capabilities.