OptPO: Optimal Rollout Allocation for Test-time Policy Optimization

TL;DR

OptPO introduces an adaptive, Bayesian-based method for test-time policy optimization that reduces computational overhead while maintaining or improving accuracy in large language models.

Contribution

It presents a novel Bayesian sequential testing framework for adaptive rollout allocation, seamlessly integrating with existing algorithms without ground-truth labels.

Findings

Significantly reduces rollout overhead in diverse benchmarks

Maintains or improves accuracy compared to fixed-sample methods

Provides a unified, statistically optimal stopping approach for test-time learning

Abstract

Test-time policy optimization enables large language models (LLMs) to adapt to distribution shifts by leveraging feedback from self-generated rollouts. However, existing methods rely on fixed-budget majority voting to estimate rewards, incurring substantial computational redundancy. We propose Optimal Rollout Allocation for Test-time Policy Optimization (OptPO), a principled framework that adaptively allocates inference budgets. By formulating the voting process as a Bayesian sequential probability ratio test, OptPO dynamically halts sampling once the posterior confidence in a consensus answer exceeds a specified threshold. Crucially, it utilizes the retained rollouts for on-policy updates, seamlessly integrating with algorithms like PPO or GRPO without requiring ground-truth labels. Across diverse reasoning benchmarks, OptPO significantly reduces rollout overhead compared to fixed-sample baselines while preserving or improving accuracy. By unifying statistically optimal stopping with test-time learning, OptPO offers a computationally efficient paradigm for test-time adaptation. The source code will be open upon acceptance at https://open-upon-acceptance.