Robust Hyperbolic Learning with Curvature-Aware Optimization

TL;DR

This paper introduces a curvature-aware optimization method for hyperbolic deep learning that enhances generalization, stability, and efficiency across various tasks by adapting the manifold curvature and constraining embeddings.

Contribution

It proposes a Riemannian AdamW optimizer and a hyperbolic scaling approach for more stable and adaptable hyperbolic learning, addressing overfitting and computational issues.

Findings

Improved performance across vision, EEG, and metric learning tasks.

Reduced runtime and increased stability in hyperbolic training.

Enhanced generalization by curvature-aware optimization.

Abstract

Hyperbolic deep learning has become a growing research direction in computer vision due to the unique properties afforded by the alternate embedding space. The negative curvature and exponentially growing distance metric provide a natural framework for capturing hierarchical relationships between datapoints and allowing for finer separability between their embeddings. However, current hyperbolic learning approaches are still prone to overfitting, computationally expensive, and prone to instability, especially when attempting to learn the manifold curvature to adapt to tasks and different datasets. To address these issues, our paper presents a derivation for Riemannian AdamW that helps increase hyperbolic generalization ability. For improved stability, we introduce a novel fine-tunable hyperbolic scaling approach to constrain hyperbolic embeddings and reduce approximation errors. Using this along with our curvature-aware learning schema for Riemannian Optimizers enables the combination of curvature and non-trivialized hyperbolic parameter learning. Our approach demonstrates consistent performance improvements across Computer Vision, EEG classification, and hierarchical metric learning tasks while greatly reducing runtime.