CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR

TL;DR

CLIPO introduces a contrastive learning approach into policy optimization for reinforcement learning with verifiable rewards, enhancing the robustness and generalization of large language models in reasoning tasks.

Contribution

It is the first to incorporate contrastive learning into RLVR, improving reasoning consistency and reducing hallucinations in LLMs during policy training.

Findings

Consistently improves RLVR baseline performance.

Enhances reasoning robustness and reduces hallucinations.

Demonstrates superior generalization across benchmarks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.