Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning

TL;DR

This paper introduces Plausible Negative Samples (PNS), a method for generating high-quality negative training data that enhances the reasoning abilities of Large Language Models by focusing on sample quality and structural coherence.

Contribution

We propose PNS, a novel approach using reverse reinforcement learning to synthesize high-quality negative samples that improve LLM reasoning performance.

Findings

PNS outperforms existing negative sample methods.

PNS achieves an average of 2.03% improvement on reasoning benchmarks.

PNS is effective as a plug-and-play data source for preference optimization.

Abstract

Learning from negative samples holds great promise for improving Large Language Model (LLM) reasoning capability, yet existing methods treat all incorrect responses as equally informative, overlooking the crucial role of sample quality. To address this, we propose Plausible Negative Samples (PNS), a method that synthesizes high-quality negative samples exhibiting expected format and structural coherence while ultimately yielding incorrect answers. PNS trains a dedicated model via reverse reinforcement learning (RL) guided by a composite reward combining format compliance, accuracy inversion, reward model assessment, and chain-of-thought evaluation, generating responses nearly indistinguishable from correct solutions. We further validate PNS as a plug-and-play data source for preference optimization across three backbone models on seven mathematical reasoning benchmarks. Results demonstrate that PNS consistently outperforms other negative sample synthesis methods, achieving an average improvement of 2.03% over RL-trained models.