Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning

TL;DR

This paper introduces A$^2$D, an adaptive ability decomposing method that improves reinforcement learning with verifiable rewards for large language models by decomposing complex questions into simpler sub-questions, enhancing reasoning.

Contribution

The paper proposes a novel A$^2$D method that decomposes questions to improve RLVR effectiveness, functioning as a plug-and-play module adaptable to various algorithms.

Findings

A$^2$D outperforms baseline methods in reasoning tasks.

The decomposer effectively guides the reasoner with sub-questions.

Analysis reveals how RLVR influences decomposer performance.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown great potential to enhance the reasoning ability of large language models (LLMs). However, due to the limited amount of information provided during the RLVR process, the model can only engage in largely blind exploration, which often results in failure on challenging problems. To provide additional information for the RLVR process without relying on a teacher model, we propose A$^2$D, an Adaptive Ability Decomposing method for enhancing the effectiveness of RLVR. Specifically, we first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions. Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance. To better understand A$^2$D, we first compare its performance with competitive baselines, showing its effectiveness. Next, we observe that our method functions as a plug-and-play module that can be applied to different RLVR algorithms. Furthermore, we conduct an analysis of the decomposer, revealing how the RLVR process affects its performance and behavior, and which type of guidance is better suited for enhancing the reasoner's exploration and exploitation abilities.