TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis

TL;DR

TTVS is a novel framework that enhances large reasoning models' test-time adaptation by dynamically generating diverse test query variations, improving performance without labeled data.

Contribution

Introduces TTVS, combining online variational synthesis and hybrid exploration to enable self-evolving LRMs using unlabeled test data.

Findings

TTVS outperforms existing test-time adaptation methods across eight model architectures.

TTVS surpasses state-of-the-art supervised RL techniques trained on labeled data.

Enforces models to learn problem logic rather than superficial patterns.

Abstract

Despite significant advances in Large Reasoning Models (LRMs) driven by reinforcement learning with verifiable rewards (RLVR), this paradigm is fundamentally limited in specialized or novel domains where such supervision is prohibitively expensive or unavailable, posing a key challenge for test-time adaptation. While existing test-time methods offer a potential solution, they are constrained by learning from static query sets, risking overfitting to textual patterns. To address this gap, we introduce Test-Time Variational Synthesis (TTVS), a novel framework that enables LRMs to self-evolve by dynamically augmenting the training stream from unlabeled test queries. TTVS comprises two synergistic modules: (1) Online Variational Synthesis, which transforms static test queries into a dynamic stream of diverse, semantically-equivalent variations, enforcing the model to learn underlying problem logic rather than superficial patterns; (2) Test-time Hybrid Exploration, which balances accuracy-driven exploitation with consistency-driven exploration across synthetic variants. Extensive experiments show TTVS yields superior performance across eight model architectures. Notably, using only unlabeled test-time data, TTVS not only surpasses other test-time adaptation methods but also outperforms state-of-the-art supervised RL-based techniques trained on vast, high-quality labeled data.