From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning

TL;DR

This paper investigates how reinforcement learning (RL) enhances reasoning capabilities, demonstrating that RL synthesizes complex strategies from atomic skills learned via supervised fine-tuning, especially when models have mastered foundational reasoning skills.

Contribution

It reveals that RL acts as a reasoning synthesizer rather than just amplifying existing behaviors, emphasizing the importance of atomic skill mastery for effective complex reasoning generalization.

Findings

RL struggles with out-of-distribution generalization, especially in zero-shot settings.

Supervised fine-tuning achieves high in-distribution accuracy but fails out-of-distribution.

RL can synthesize complex reasoning strategies if atomic skills are mastered beforehand.

Abstract

The mechanism by which RL contributes to reasoning capabilities-whether it incentivizes the synthesis of new skills or merely amplifies existing behaviors-remains a subject of intense debate. In this work, we investigate this question through the lens of Complementary Reasoning, a complex task that requires integrating internal parametric knowledge with external contextual information. Using a controlled synthetic dataset of human biographies, we strictly decouple this ability into two atomic skills: Parametric Reasoning (relying on internal knowledge) and Contextual Reasoning (depending on external information). To rigorously assess capability boundaries, we evaluate generalization across three distinct levels of difficulty: I.I.D., Composition, and Zero-shot settings. We find that while SFT is sufficient for in-distribution performance, it struggles with O.O.D. generalization, particularly in Zero-shot settings where relational combinations are novel. Crucially, we identify the SFT Generalization Paradox: Models supervised solely on the composite task achieve near-perfect in-distribution accuracy but collapse on out-of-distribution generalization, indicating their reliance on rote memorization of path shortcuts. In contrast, we find that RL acts as a reasoning synthesizer rather than a probability amplifier. However, we uncover a strict atomic prerequisite: RL can only synthesize these complex strategies if the base model has first mastered the independent atomic skills (Parametric and Contextual) via SFT. These findings challenge the view of RL as a mere amplifier, suggesting that given sufficient atomic foundations, RL can actively synthesize complex reasoning strategies from learned primitives without explicit supervision on such complex strategies. This indicates that decoupled atomic training followed by RL offers a scalable path to generalization for complex reasoning tasks.