The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVR

TL;DR

This paper demonstrates that training only a small, randomly selected subset of parameters in RLVR can match or outperform full-model finetuning, revealing many viable sparse subnetworks within pretrained models.

Contribution

It introduces the Multiple Ticket Hypothesis, showing that random sparse subnetworks can effectively fine-tune RLVR models, highlighting inherent redundancy and low-dimensional update constraints.

Findings

Training 1% of parameters matches or exceeds full model performance.

Different random sparse masks have minimal overlap yet all succeed.

Implicit KL constraints in RLVR enable arbitrary sparse subnetworks to perform well.

Abstract

The Lottery Ticket Hypothesis demonstrated that sparse subnetworks can match full-model performance, suggesting parameter redundancy. Meanwhile, in Reinforcement Learning with Verifiable Rewards (RLVR), recent work has shown that updates concentrate on a sparse subset of parameters, which further lends evidence to this underlying redundancy. We study the simplest possible way to exploit this redundancy: training only a randomly selected subset of parameters at extreme sparsities. Empirically, we find that training just 1\% of parameters matches or exceeds full-parameter RLVR finetuning across 3 models and 2 task domains. Moreover, different random masks show minimal overlap ($\leq 0.005$ Jaccard similarity) and yet all succeed, suggesting pretrained models contain many viable sparse subnetworks rather than one privileged set. We term this the Multiple Ticket Hypothesis. We explain this phenomenon through the implicit per-step KL constraint in RLVR, which restricts updates to a low-dimensional subspace, enabling arbitrary sparse masks to succeed.