Learning Structured Representations with Hyperbolic Embeddings

TL;DR

This paper introduces HypStructure, a hyperbolic regularization method that effectively embeds hierarchical label structures into learned representations, improving generalization and OOD detection in vision tasks.

Contribution

It proposes a novel hyperbolic regularizer for hierarchical embedding that enhances representation quality and performance across vision benchmarks.

Findings

Reduces distortion in hierarchical embeddings

Improves generalization especially in low-dimensional settings

Enhances out-of-distribution detection capabilities

Abstract

Most real-world datasets consist of a natural hierarchy between classes or an inherent label structure that is either already available or can be constructed cheaply. However, most existing representation learning methods ignore this hierarchy, treating labels as permutation invariant. Recent work [Zeng et al., 2022] proposes using this structured information explicitly, but the use of Euclidean distance may distort the underlying semantic context [Chen et al., 2013]. In this work, motivated by the advantage of hyperbolic spaces in modeling hierarchical relationships, we propose a novel approach HypStructure: a Hyperbolic Structured regularization approach to accurately embed the label hierarchy into the learned representations. HypStructure is a simple-yet-effective regularizer that consists of a hyperbolic tree-based representation loss along with a centering loss, and can be combined with any standard task loss to learn hierarchy-informed features. Extensive experiments on several large-scale vision benchmarks demonstrate the efficacy of HypStructure in reducing distortion and boosting generalization performance especially under low dimensional scenarios. For a better understanding of structured representation, we perform eigenvalue analysis that links the representation geometry to improved Out-of-Distribution (OOD) detection performance seen empirically. The code is available at \url{https://github.com/uiuctml/HypStructure}.