Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learning

TL;DR

This paper introduces Geometric Action Control (GAC), a new method for continuous reinforcement learning that preserves the geometry of action spaces, simplifies computation, and outperforms existing approaches on multiple benchmarks.

Contribution

GAC offers a geometrically grounded, computationally efficient alternative to Gaussian and vMF policies, improving performance and simplicity in continuous RL control tasks.

Findings

GAC matches or exceeds state-of-the-art performance on MuJoCo benchmarks.

GAC reduces parameter count and computational complexity compared to vMF distributions.

Ablation studies confirm the importance of spherical normalization and adaptive concentration control.

Abstract

Gaussian policies have dominated continuous control in deep reinforcement learning (RL), yet they suffer from a fundamental mismatch: their unbounded support requires ad-hoc squashing functions that distort the geometry of bounded action spaces. While von Mises-Fisher (vMF) distributions offer a theoretically grounded alternative on the sphere, their reliance on Bessel functions and rejection sampling hinders practical adoption. We propose \textbf{Geometric Action Control (GAC)}, a novel action generation paradigm that preserves the geometric benefits of spherical distributions while \textit{simplifying computation}. GAC decomposes action generation into a direction vector and a learnable concentration parameter, enabling efficient interpolation between deterministic actions and uniform spherical noise. This design reduces parameter count from \(2d\) to \(d+1\), and avoids the \(O(dk)\) complexity of vMF rejection sampling, achieving simple \(O(d)\) operations. Empirically, GAC consistently matches or exceeds state-of-the-art methods across six MuJoCo benchmarks, achieving 37.6\% improvement over SAC on Ant-v4 and up to 112\% on complex DMControl tasks, demonstrating strong performance across diverse benchmarks. Our ablation studies reveal that both \textbf{spherical normalization} and \textbf{adaptive concentration control} are essential to GAC's success. These findings suggest that robust and efficient continuous control does not require complex distributions, but a principled respect for the geometry of action spaces.