Fast and Geometrically Grounded Lorentz Neural Networks

TL;DR

This paper introduces a new Lorentz linear layer for hyperbolic neural networks that maintains hyperbolic properties during training, enabling efficient and geometrically grounded learning in hyperbolic space.

Contribution

The authors propose a novel Lorentz linear layer based on the distance-to-hyperplane formulation, improving the scaling of hyperbolic norms during training.

Findings

The new Lorentz linear layer ensures linear scaling of hyperbolic norms.

Algorithmic efficiencies enable hyperbolic networks to match Euclidean computation speeds.

The approach preserves hyperbolic geometry properties throughout training.

Abstract

Hyperbolic space is quickly gaining traction as a promising geometry for hierarchical and robust representation learning. A core open challenge is the development of a mathematical formulation of hyperbolic neural networks that is both efficient and captures the key properties of hyperbolic space. The Lorentz model of hyperbolic space has been shown to enable both fast forward and backward propagation. However, we prove that, with the current formulation of Lorentz linear layers, the hyperbolic norms of the outputs scale logarithmically with the number of gradient descent steps, nullifying the key advantage of hyperbolic geometry. We propose a new Lorentz linear layer grounded in the well-known ``distance-to-hyperplane" formulation. We prove that our formulation results in the usual linear scaling of output hyperbolic norms with respect to the number of gradient descent steps. Our new formulation, together with further algorithmic efficiencies through Lorentzian activation functions and a new caching strategy results in neural networks fully abiding by hyperbolic geometry while simultaneously bridging the computation gap to Euclidean neural networks. Code available at: https://github.com/robertdvdk/hyperbolic-fully-connected.