Lorentzian Residual Neural Networks

TL;DR

This paper introduces LResNet, a Lorentzian residual neural network that improves stability and efficiency in hyperbolic neural networks, demonstrating superior performance on graph and vision tasks.

Contribution

The paper presents a novel Lorentzian residual neural network that addresses limitations of existing hyperbolic residual networks, enabling more stable and efficient hierarchical data modeling.

Findings

LResNet outperforms state-of-the-art Euclidean and hyperbolic models.

The method improves stability and efficiency in hyperbolic neural networks.

LResNet is applicable to CNNs, GNNs, and graph Transformers.

Abstract

Hyperbolic neural networks have emerged as a powerful tool for modeling hierarchical data structures prevalent in real-world datasets. Notably, residual connections, which facilitate the direct flow of information across layers, have been instrumental in the success of deep neural networks. However, current methods for constructing hyperbolic residual networks suffer from limitations such as increased model complexity, numerical instability, and errors due to multiple mappings to and from the tangent space. To address these limitations, we introduce LResNet, a novel Lorentzian residual neural network based on the weighted Lorentzian centroid in the Lorentz model of hyperbolic geometry. Our method enables the efficient integration of residual connections in Lorentz hyperbolic neural networks while preserving their hierarchical representation capabilities. We demonstrate that our method can theoretically derive previous methods while offering improved stability, efficiency, and effectiveness. Extensive experiments on both graph and vision tasks showcase the superior performance and robustness of our method compared to state-of-the-art Euclidean and hyperbolic alternatives. Our findings highlight the potential of LResNet for building more expressive neural networks in hyperbolic embedding space as a generally applicable method to multiple architectures, including CNNs, GNNs, and graph Transformers.