Hyperbolic Residual Quantization: Discrete Representations for Data with Latent Hierarchies

TL;DR

This paper introduces Hyperbolic Residual Quantization (HRQ), a novel method that embeds data in hyperbolic space to better capture hierarchical structures in discrete representations, outperforming traditional Euclidean-based methods.

Contribution

The paper proposes HRQ, which adapts residual quantization to hyperbolic geometry, providing a more natural fit for hierarchical data and improving downstream task performance.

Findings

HRQ outperforms Euclidean RQ by up to 20% in hierarchy modeling tasks.

Embedding in hyperbolic space better captures latent hierarchical structures.

HRQ improves downstream task performance on hierarchical data.

Abstract

Hierarchical data arise in countless domains, from biological taxonomies and organizational charts to legal codes and knowledge graphs. Residual Quantization (RQ) is widely used to generate discrete, multitoken representations for such data by iteratively quantizing residuals in a multilevel codebook. However, its reliance on Euclidean geometry can introduce fundamental mismatches that hinder modeling of hierarchical branching, necessary for faithful representation of hierarchical data. In this work, we propose Hyperbolic Residual Quantization (HRQ), which embeds data natively in a hyperbolic manifold and performs residual quantization using hyperbolic operations and distance metrics. By adapting the embedding network, residual computation, and distance metric to hyperbolic geometry, HRQ imparts an inductive bias that aligns naturally with hierarchical branching. We claim that HRQ in comparison to RQ can generate more useful for downstream tasks discrete hierarchical representations for data with latent hierarchies. We evaluate HRQ on two tasks: supervised hierarchy modeling using WordNet hypernym trees, where the model is supervised to learn the latent hierarchy - and hierarchy discovery, where, while latent hierarchy exists in the data, the model is not directly trained or evaluated on a task related to the hierarchy. Across both scenarios, HRQ hierarchical tokens yield better performance on downstream tasks compared to Euclidean RQ with gains of up to $20\%$ for the hierarchy modeling task. Our results demonstrate that integrating hyperbolic geometry into discrete representation learning substantially enhances the ability to capture latent hierarchies.