ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning

TL;DR

ECHO introduces an adaptive hybrid optimization method for test-time reinforcement learning that effectively manages exploration and reduces collapse by leveraging entropy and confidence measures, leading to improved performance and robustness.

Contribution

The paper proposes ECHO, a novel approach that adaptively controls rollout branching and pruning using entropy and confidence, addressing key challenges in test-time reinforcement learning.

Findings

ECHO outperforms existing methods on multiple reasoning benchmarks.

ECHO demonstrates better generalization with limited rollout budgets.

ECHO enhances training robustness by mitigating early-stage bias.

Abstract

Test-time reinforcement learning generates multiple candidate answers via repeated rollouts and performs online updates using pseudo-labels constructed by majority voting. To reduce overhead and improve exploration, prior work introduces tree structured rollouts, which share reasoning prefixes and branch at key nodes to improve sampling efficiency. However, this paradigm still faces two challenges: (1) high entropy branching can trigger rollout collapse, where the branching budget concentrates on a few trajectories with consecutive high-entropy segments, rapidly reducing the number of effective branches; (2) early pseudo-labels are noisy and biased, which can induce self-reinforcing overfitting, causing the policy to sharpen prematurely and suppress exploration. To address these issues, we propose Entropy Confidence Hybrid Group Relative Policy Optimization (ECHO). During rollout, ECHO jointly leverages local entropy and group level confidence to adaptively control branch width, and further introduces online confidence-based pruning to terminate persistently low confidence branches, avoiding high entropy traps and mitigating collapse. During policy updates, ECHO employs confidence adaptive clipping and an entropy confidence hybrid advantage shaping approach to enhance training robustness and mitigate early stage bias. Experiments demonstrate that ECHO achieves consistent gains on multiple mathematical and visual reasoning benchmarks, and generalizes more effectively under a limited rollout budget.