Tailored Primitive Initialization is the Secret Key to Reinforcement Learning

TL;DR

This paper introduces Tailor, a finetuning pipeline that discovers and curates reasoning primitives to improve the initialization of language models, leading to more efficient and stable reinforcement learning for reasoning tasks.

Contribution

The paper proposes a novel method for initializing language models with diverse reasoning primitives, enhancing RL training efficiency and stability.

Findings

Tailor improves reasoning token coverage in LLMs.

Enhanced initialization leads to higher RL performance on reasoning benchmarks.

Diverse reasoning primitives result in more sample-efficient RL training.

Abstract

Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). While RL has demonstrated substantial performance gains, it still faces key challenges, including low sampling efficiency and a strong dependence on model initialization: some models achieve rapid improvements with minimal RL steps, while others require significant training data to make progress. In this work, we investigate these challenges through the lens of reasoning token coverage and argue that initializing LLMs with diverse, high-quality reasoning primitives is essential for achieving stable and sample-efficient RL training. We propose Tailor, a finetuning pipeline that automatically discovers and curates novel reasoning primitives, thereby expanding the coverage of reasoning-state distributions before RL. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that Tailor generates more diverse and higher-quality warm-start data, resulting in higher downstream RL performance.