Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?

TL;DR

This paper investigates whether reinforcement post training (RPT) improvements in large language models transfer effectively to unseen domains, revealing that gains are inconsistent and often diminish outside the fine-tuning data.

Contribution

The study provides the first comprehensive analysis of RPT's domain transferability, highlighting its limitations in generalizing reasoning improvements across diverse domains.

Findings

RPT improves performance on similar domains

Gains often diminish on different reasoning patterns

Transferability of RPT is inconsistent across domains

Abstract

Reinforcement post training (RPT) has recently shown promise in improving the reasoning abilities of large language models (LLMs). However, it remains unclear how well these improvements generalize to new domains, as prior work evaluates RPT models on data from the same domains used for post-training. To understand the generalizability of RPT, we conduct two studies with specific focus on Reinforcement Learning with Verifiable Rewards (RLVR). (1) Observational: we compare a wide range of open-weight RPT models against their corresponding base models across multiple domains, including both seen and unseen domains in their fine-tuning data. (2) Interventional: we fine-tune LLMs with RPT on single domains and evaluate their performance across multiple domains. Both studies converge on the same conclusion that, although RPT brings substantial gains on tasks similar to the fine-tuning data, the gains generalize inconsistently and can vanish on domains with different reasoning patterns.