Why are hyperbolic neural networks effective? A study on hierarchical representation capability

TL;DR

This paper investigates why hyperbolic neural networks are effective, revealing they do not achieve optimal hierarchical embedding, and proposes strategies to enhance their hierarchical representation capability and downstream performance.

Contribution

The study provides a benchmark for evaluating hierarchical representation capability in HNNs and introduces pre-training strategies to improve their effectiveness.

Findings

HNNs do not reach theoretical optimal embedding.

Hierarchical Representation Capability is influenced by objectives and structure.

Pre-training strategies significantly boost HNN performance.

Abstract

Hyperbolic Neural Networks (HNNs), operating in hyperbolic space, have been widely applied in recent years, motivated by the existence of an optimal embedding in hyperbolic space that can preserve data hierarchical relationships (termed Hierarchical Representation Capability, HRC) more accurately than Euclidean space. However, there is no evidence to suggest that HNNs can achieve this theoretical optimal embedding, leading to much research being built on flawed motivations. In this paper, we propose a benchmark for evaluating HRC and conduct a comprehensive analysis of why HNNs are effective through large-scale experiments. Inspired by the analysis results, we propose several pre-training strategies to enhance HRC and improve the performance of downstream tasks, further validating the reliability of the analysis. Experiments show that HNNs cannot achieve the theoretical optimal embedding. The HRC is significantly affected by the optimization objectives and hierarchical structures, and enhancing HRC through pre-training strategies can significantly improve the performance of HNNs.