Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL

TL;DR

This paper introduces a hierarchical RL framework called Introspective LLM that learns to adapt the sampling temperature during language generation, improving reasoning performance and interpretability over static or heuristic methods.

Contribution

It presents a novel method for learning dynamic temperature control in LLMs using hierarchical RL, directly optimizing for downstream rewards.

Findings

Learned temperature policies outperform fixed baselines.

Temperature control aligns with reasoning uncertainty.

Enhanced exploration improves mathematical reasoning results.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) trains large language models (LLMs) from sampled trajectories, making decoding strategy a core component of learning rather than a purely inference-time choice. Sampling temperature directly controls the exploration--exploitation trade-off by modulating policy entropy, yet existing methods rely on static values or heuristic adaptations that are decoupled from task-level rewards. We propose Introspective LLM, a hierarchical reinforcement learning framework that learns to control sampling temperature during generation. At each decoding step, the model selects a temperature based on its hidden state and samples the next token from the resulting distribution. Temperature and token policies are jointly optimized from downstream rewards using a coordinate ascent scheme. Experiments on mathematical reasoning benchmarks show that learned temperature policies outperform fixed and heuristic baselines, while exhibiting interpretable exploration behaviors aligned with reasoning uncertainty.