Hierarchical Budget Policy Optimization for Adaptive Reasoning

TL;DR

This paper introduces HBPO, a reinforcement learning framework that enables large reasoning models to adaptively determine their reasoning depth, significantly improving efficiency and accuracy across reasoning tasks.

Contribution

HBPO is a novel hierarchical training method that allows models to learn problem-specific reasoning depths without sacrificing capability, addressing efficiency and exploration challenges.

Findings

Reduces token usage by up to 60.6%.

Improves accuracy by 3.14% on reasoning benchmarks.

Models exhibit emergent adaptive reasoning behavior.

Abstract

Large reasoning models achieve remarkable performance through extensive chain-of-thought generation, yet they suffer from a critical inefficiency: applying uniformly extensive reasoning regardless of problem complexity. We present Hierarchical Budget Policy Optimization (HBPO), a reinforcement learning framework that enables models to learn problem-specific reasoning depths without sacrificing capability. Unlike existing approaches that impose rigid constraints or rely on discrete mode selection, HBPO partitions the exploration space into budget-constrained hierarchies (512-2560 tokens), each with differentiated reward structures that preserve both efficiency incentives and reasoning capabilities. This design addresses a fundamental challenge in efficient reasoning training: traditional length penalties systematically bias models away from necessary long reasoning paths, causing exploration space collapse. Through hierarchical sampling and budget-aware rewards, HBPO maintains exploration diversity while teaching models to recognize when extended deliberation is warranted. Extensive experiments demonstrate that HBPO reduces average token usage by up to 60.6% while improving accuracy by 3.14% across four reasoning benchmarks. Most notably, HBPO exhibits emergent adaptive behavior where models automatically adjust reasoning depth based on problem complexity. Our results suggest that reasoning efficiency and capability are not inherently conflicting, and can be simultaneously optimized through appropriately structured hierarchical training that preserves exploration diversity.