Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning

TL;DR

This paper introduces TAMPO, a novel framework that learns to adapt the temperature hyperparameter in LLM reinforcement learning dynamically, improving exploration and policy performance without extra rollouts.

Contribution

The paper presents TAMPO, a hierarchical meta-policy approach that adaptively controls temperature in LLM RL, outperforming fixed or heuristic schedules on reasoning benchmarks.

Findings

TAMPO outperforms fixed temperature baselines on five reasoning tasks.

Adaptive temperature control improves policy learning efficiency.

Meta-policy effectively learns temperature schedules without additional rollouts.

Abstract

Temperature is a crucial hyperparameter in large language models (LLMs), controlling the trade-off between exploration and exploitation during text generation. High temperatures encourage diverse but noisy outputs, while low temperatures produce focused outputs but may cause premature convergence. Yet static or heuristic temperature schedules fail to adapt to the dynamic demands of reinforcement learning (RL) throughout training, often limiting policy improvement. We propose Temperature Adaptive Meta Policy Optimization (TAMPO), a new framework that recasts temperature control as a learnable meta-policy. TAMPO operates through a hierarchical two-loop process. In the inner loop, the LLM policy is updated (e.g., using GRPO) with trajectories sampled at the temperature selected by the meta-policy. In the outer loop, meta-policy updates the distribution over candidate temperatures by rewarding those that maximize the likelihood of high-advantage trajectories. This trajectory-guided, reward-driven mechanism enables online adaptation without additional rollouts, directly aligning exploration with policy improvement. On five mathematical reasoning benchmarks, TAMPO outperforms baselines using fixed or heuristic temperatures, establishing temperature as an effective learnable meta-policy for adaptive exploration in LLM reinforcement learning. Accepted at ICLR 2026.