Understanding and Improving Hyperbolic Deep Reinforcement Learning

TL;DR

This paper investigates the challenges of hyperbolic deep reinforcement learning, analyzes gradient instability issues, and introduces Hyper++, a new method that improves stability and performance in hyperbolic RL tasks.

Contribution

We identify key factors causing training failures in hyperbolic deep RL and propose Hyper++, a novel approach with regularization and stable training techniques.

Findings

Hyper++ achieves more stable learning on ProcGen and Atari-5.

It outperforms prior hyperbolic RL agents in efficiency and effectiveness.

Reduces wall-clock time by approximately 30%.

Abstract

The exponential volume growth of hyperbolic geometry can embed the hierarchical relationships between states in reinforcement learning (RL) with far less distortion than Euclidean space. However, hyperbolic deep RL faces severe optimization challenges, and formal analysis of why optimization fails is lacking. We identify key factors that determine the success and failure of training hyperbolic deep RL agents. By analyzing the gradients of core operations in the Poincar\'e Ball and Hyperboloid models of hyperbolic geometry, we show that large-norm embeddings destabilize gradient-based training, leading to trust-region violations in proximal policy optimization (PPO). Based on these insights, we introduce Hyper++, a new hyperbolic deep RL agent that consists of three components: (1) feature regularization guaranteeing bounded norms while avoiding the curse of dimensionality from clipping; (2) a categorical value loss for stable critic training; and (3) a more optimization-friendly formulation of hyperbolic network layers. On ProcGen, we show that Hyper++ guarantees stable learning, outperforms prior hyperbolic agents, and reduces wall-clock time by approximately 30%. On Atari-5 with Double DQN, Hyper++ strongly outperforms Euclidean and hyperbolic baselines. We release our code at https://github.com/Probabilistic-and-Interactive-ML/hyper-rl.