Decomposing Elements of Problem Solving: What "Math" Does RL Teach?

TL;DR

This paper investigates how reinforcement learning improves mathematical reasoning in large language models by decomposing problem-solving into planning, execution, and verification, revealing RL's strengths and limitations in skill development.

Contribution

It introduces a decomposition framework for problem-solving skills and demonstrates RL's impact on execution robustness and planning limitations through empirical and synthetic experiments.

Findings

RL enhances execution robustness but struggles with new problems

Models mainly improve execution, not planning skills

Synthetic tasks confirm RL's role and limitations in exploration

Abstract

Mathematical reasoning tasks have become prominent benchmarks for assessing the reasoning capabilities of LLMs, especially with reinforcement learning (RL) methods such as GRPO showing significant performance gains. However, accuracy metrics alone do not support fine-grained assessment of capabilities and fail to reveal which problem-solving skills have been internalized. To better understand these capabilities, we propose to decompose problem solving into fundamental capabilities: Plan (mapping questions to sequences of steps), Execute (correctly performing solution steps), and Verify (identifying the correctness of a solution). Empirically, we find that GRPO mainly enhances the execution skill-improving execution robustness on problems the model already knows how to solve-a phenomenon we call temperature distillation. More importantly, we show that RL-trained models struggle with fundamentally new problems, hitting a 'coverage wall' due to insufficient planning skills. To explore RL's impact more deeply, we construct a minimal, synthetic solution-tree navigation task as an analogy for mathematical problem-solving. This controlled setup replicates our empirical findings, confirming RL primarily boosts execution robustness. Importantly, in this setting, we identify conditions under which RL can potentially overcome the coverage wall through improved exploration and generalization to new solution paths. Our findings provide insights into the role of RL in enhancing LLM reasoning, expose key limitations, and suggest a path toward overcoming these barriers. Code is available at https://github.com/cfpark00/RL-Wall.