Trust Regions Sell, But Who's Buying? Overlap Geometry as an Alternative Trust Region for Policy Optimization

TL;DR

This paper introduces overlap geometry as a new trust region for policy optimization, using the Bhattacharyya coefficient to better control likelihood-ratio excursions and improve training stability over traditional KL-based methods.

Contribution

It proposes overlap geometry as an alternative to KL divergence for trust regions, deriving BTRPO and BPPO algorithms that enhance robustness and performance.

Findings

Overlap-based updates improve robustness in policy training.

Overlap constraints outperform KL-based methods in stability.

Proposed methods achieve better aggregate performance.

Abstract

Standard trust-region methods constrain policy updates via Kullback-Leibler (KL) divergence. However, KL controls only an average divergence and does not directly prevent rare, large likelihood-ratio excursions that destabilize training--precisely the failure mode that motivates heuristics such as PPO's clipping. We propose overlap geometry as an alternative trust region, constraining distributional overlap via the Bhattacharyya coefficient (closely related to the Hellinger/Renyi-1/2 geometry). This objective penalizes separation in the ratio tails, yielding tighter control over likelihood-ratio excursions without relying on total variation bounds that can be loose in tail regimes. We derive Bhattacharyya-TRPO (BTRPO) and Bhattacharyya-PPO (BPPO), enforcing overlap constraints via square-root ratio updates: BPPO clips the square-root ratio q = sqrt(r), and BTRPO applies a quadratic Hellinger/Bhattacharyya penalty. Empirically, overlap-based updates improve robustness and aggregate performance as measured by RLiable under matched training budgets, suggesting overlap constraints as a practical, principled alternative to KL for stable policy optimization.