Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers

TL;DR

This paper introduces clipped hyperbolic neural networks that overcome previous limitations by addressing vanishing gradients, resulting in classifiers that outperform traditional hyperbolic models and match Euclidean neural networks across various datasets.

Contribution

The paper proposes a simple clipping technique during training to enhance hyperbolic neural networks, making them competitive with Euclidean models on standard benchmarks.

Findings

Clipped HNNs outperform unclipped HNNs on hierarchical datasets.

Clipped HNNs match Euclidean neural networks on MNIST, CIFAR, and ImageNet.

Clipped HNNs exhibit improved adversarial robustness and out-of-distribution detection.

Abstract

Hyperbolic space can naturally embed hierarchies, unlike Euclidean space. Hyperbolic Neural Networks (HNNs) exploit such representational power by lifting Euclidean features into hyperbolic space for classification, outperforming Euclidean neural networks (ENNs) on datasets with known semantic hierarchies. However, HNNs underperform ENNs on standard benchmarks without clear hierarchies, greatly restricting HNNs' applicability in practice. Our key insight is that HNNs' poorer general classification performance results from vanishing gradients during backpropagation, caused by their hybrid architecture connecting Euclidean features to a hyperbolic classifier. We propose an effective solution by simply clipping the Euclidean feature magnitude while training HNNs. Our experiments demonstrate that clipped HNNs become super-hyperbolic classifiers: They are not only consistently better than HNNs which already outperform ENNs on hierarchical data, but also on-par with ENNs on MNIST, CIFAR10, CIFAR100 and ImageNet benchmarks, with better adversarial robustness and out-of-distribution detection.