Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning

TL;DR

This paper introduces a novel contrastive learning framework for hyperbolic graph embeddings that addresses dimensional collapse and effectively captures hierarchical data, improving the quality of learned representations.

Contribution

It proposes a new contrastive learning method with specialized alignment and uniformity metrics tailored for hyperbolic space, mitigating dimensional collapse in hierarchical graph embeddings.

Findings

The method effectively prevents hyperbolic dimensional collapse.

It captures hierarchical information more accurately.

Experimental results show improved embedding quality.

Abstract

Learning generalizable self-supervised graph representations for downstream tasks is challenging. To this end, Contrastive Learning (CL) has emerged as a leading approach. The embeddings of CL are arranged on a hypersphere where similarity is measured by the cosine distance. However, many real-world graphs, especially of hierarchical nature, cannot be embedded well in the Euclidean space. Although the hyperbolic embedding is suitable for hierarchical representation learning, naively applying CL to the hyperbolic space may result in the so-called dimension collapse, i.e., features will concentrate mostly within few density regions, leading to poor utilization of the whole feature space. Thus, we propose a novel contrastive learning framework to learn high-quality graph embeddings in hyperbolic space. Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information, as well as we propose a substitute of the uniformity metric to prevent the so-called dimensional collapse. We show that in the hyperbolic space one has to address the leaf- and height-level uniformity related to properties of trees. In the ambient space of the hyperbolic manifold these notions translate into imposing an isotropic ring density towards boundaries of Poincar\'e ball. Our experiments support the efficacy of our method.