PACEvolve++: Improving Test-time Learning for Evolutionary Search Agents

TL;DR

PACEvolve++ introduces a reinforcement learning framework that enables test-time policy adaptation for evolutionary search agents, improving convergence speed and stability across various tasks.

Contribution

It proposes a novel advisor-model RL framework with phase-adaptive training to enhance evolutionary search performance.

Findings

Outperforms state-of-the-art frameworks in convergence speed.

Stabilizes test-time training during evolutionary search.

Effective across diverse tasks like load balancing and protein fitness.

Abstract

Large language models have become drivers of evolutionary search, but most systems rely on a fixed, prompt-elicited policy to sample next candidates. This limits adaptation in practical engineering and research tasks, where evaluations are expensive, and progress depends on learning task-specific search dynamics. We introduce PACEvolve++, an advisor-model reinforcement learning framework for test-time policy adaptation in evolutionary search agents. PACEvolve++ decouples strategic search decisions from implementation: a trainable advisor generates, assesses, and selects hypotheses, while a stronger frontier model translates selected hypotheses into executable candidates. To train the advisor under non-stationary feedback, we propose a phase-adaptive approach that adapts its optimization strategy to different phases of the evolutionary process. Early in evolution, it uses group-relative feedback to learn broad search preferences; later, as reward gaps compress, it emphasizes best-of-$k$ frontier contribution to support stable refinement. Across expert-parallel load balancing, sequential recommendation, and protein fitness extrapolation, PACEvolve++ outperforms the state-of-the-art evolutionary search framework with frontier models, achieving faster convergence and stabilizing test-time training during evolutionary search.