The Unlearnability Phenomenon in RLVR for Language Models

TL;DR

This paper investigates the unlearnability phenomenon in RLVR for language models, revealing fundamental limitations in current approaches and characterizing unlearnable data through gradient analysis.

Contribution

It systematically characterizes unlearnable data in RLVR training and uncovers fundamental representation issues affecting learning.

Findings

Unlearnable examples have low gradient similarity with other data.

Representation flaws hinder learning even with correct rollouts.

Data augmentation does not improve gradient similarity for unlearnable data.

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model's (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks. Code and data are available at \url{https://github.com/yulinchen99/unlearnability-rlvr}.