Curvature Learning for Generalization of Hyperbolic Neural Networks

TL;DR

This paper develops a theoretical understanding of how curvature affects hyperbolic neural networks' generalization and introduces a novel curvature learning method to enhance their performance across various tasks.

Contribution

It derives a PAC-Bayesian generalization bound for HNNs related to curvature and proposes a sharpness-aware curvature learning approach to improve generalization.

Findings

Theoretical link between curvature and HNN generalization.

Proposed method improves HNN performance on multiple tasks.

Bounded approximation error and convergence of the optimization algorithm.

Abstract

Hyperbolic neural networks (HNNs) have demonstrated notable efficacy in representing real-world data with hierarchical structures via exploiting the geometric properties of hyperbolic spaces characterized by negative curvatures. Curvature plays a crucial role in optimizing HNNs. Inappropriate curvatures may cause HNNs to converge to suboptimal parameters, degrading overall performance. So far, the theoretical foundation of the effect of curvatures on HNNs has not been developed. In this paper, we derive a PAC-Bayesian generalization bound of HNNs, highlighting the role of curvatures in the generalization of HNNs via their effect on the smoothness of the loss landscape. Driven by the derived bound, we propose a sharpness-aware curvature learning method to smooth the loss landscape, thereby improving the generalization of HNNs. In our method, we design a scope sharpness measure for curvatures, which is minimized through a bi-level optimization process. Then, we introduce an implicit differentiation algorithm that efficiently solves the bi-level optimization by approximating gradients of curvatures. We present the approximation error and convergence analyses of the proposed method, showing that the approximation error is upper-bounded, and the proposed method can converge by bounding gradients of HNNs. Experiments on four settings: classification, learning from long-tailed data, learning from noisy data, and few-shot learning show that our method can improve the performance of HNNs.