Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization

TL;DR

This paper introduces Consensus Aggregation for Policy Optimization (CAPO), a method that improves policy training by aggregating multiple PPO replicas to enhance trust region compliance and performance without increasing environment interactions.

Contribution

The paper proposes CAPO, a novel aggregation technique that shifts focus from deeper to wider policy optimization, with theoretical guarantees and empirical improvements over PPO.

Findings

CAPO outperforms PPO and deeper baselines by up to 8.6x on control tasks.

Aggregation in natural parameter space achieves better trust region compliance.

Wider optimization with consensus improves sample efficiency and policy quality.

Abstract

Proximal policy optimization (PPO) approximates the trust region update using multiple epochs of clipped SGD. Each epoch may drift further from the natural gradient direction, creating path-dependent noise. To understand this drift, we can use Fisher information geometry to decompose policy updates into signal (the natural gradient projection) and waste (the Fisher-orthogonal residual that consumes trust region budget without first-order surrogate improvement). Empirically, signal saturates but waste grows with additional epochs, creating an optimization-depth dilemma. We propose Consensus Aggregation for Policy Optimization (CAPO), which redirects compute from depth to width: $K$ PPO replicates are optimized on the same batch, differing only in minibatch shuffling order, and then aggregated into a consensus. We study aggregation in two spaces: Euclidean parameter space, and the natural parameter space of the policy distribution via the logarithmic opinion pool. In natural parameter space, the consensus provably achieves higher KL-penalized surrogate and tighter trust region compliance than the mean expert; parameter averaging inherits these guarantees approximately. On continuous control tasks, CAPO outperforms PPO and compute-matched deeper baselines under fixed sample budgets by up to 8.6x. CAPO demonstrates that policy optimization can be improved by optimizing wider, rather than deeper, without additional environment interactions.