When Is Compositional Reasoning Learnable from Verifiable Rewards?

TL;DR

This paper provides a theoretical framework to understand when compositional reasoning can be learned through reinforcement learning with verifiable rewards, highlighting the importance of task-advantage ratios.

Contribution

It introduces the task-advantage ratio as a key metric for determining learnability of compositional problems in RLVR and analyzes conditions for successful learning.

Findings

Compositional problems with clear intermediate advantages are learnable.

Absence of structural advantage can lead to suboptimal convergence.

Base model quality influences the presence of advantages and learning outcomes.

Abstract

The emergence of compositional reasoning in large language models through reinforcement learning with verifiable rewards (RLVR) has been a key driver of recent empirical successes. Despite this progress, it remains unclear which compositional problems are learnable in this setting using outcome-level feedback alone. In this work, we theoretically study the learnability of compositional problems in autoregressive models under RLVR training. We identify a quantity that we call the task-advantage ratio, a joint property of the compositional problem and the base model, that characterizes which tasks and compositions are learnable from outcome-level feedback. On the positive side, using this characterization, we show that compositional problems where correct intermediate steps provide a clear advantage are efficiently learnable with RLVR. We also analyze how such an advantage naturally arises in different problems. On the negative side, when the structural advantage is not present, RLVR may converge to suboptimal compositions. We prove that, in some cases, the quality of the base model determines if such an advantage exists and whether RLVR will converge to a suboptimal solution. We hope our analysis can provide a principled theoretical understanding of when and why RLVR succeeds and when it does not.