Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning

TL;DR

This paper introduces a hyperbolic geometry-based approach for backward-compatible representation learning, effectively capturing model uncertainty and ensuring consistency across model updates.

Contribution

It proposes lifting embeddings into hyperbolic space and constraining updates within entailment cones, improving compatibility and robustness over Euclidean methods.

Findings

Hyperbolic space better models uncertainty in embeddings.

The method outperforms Euclidean-based compatibility approaches.

Robust contrastive loss enhances alignment accuracy.

Abstract

Backward compatible representation learning enables updated models to integrate seamlessly with existing ones, avoiding to reprocess stored data. Despite recent advances, existing compatibility approaches in Euclidean space neglect the uncertainty in the old embedding model and force the new model to reconstruct outdated representations regardless of their quality, thereby hindering the learning process of the new model. In this paper, we propose to switch perspectives to hyperbolic geometry, where we treat time as a natural axis for capturing a model's confidence and evolution. By lifting embeddings into hyperbolic space and constraining updated embeddings to lie within the entailment cone of the old ones, we maintain generational consistency across models while accounting for uncertainties in the representations. To further enhance compatibility, we introduce a robust contrastive alignment loss that dynamically adjusts alignment weights based on the uncertainty of the old embeddings. Experiments validate the superiority of the proposed method in achieving compatibility, paving the way for more resilient and adaptable machine learning systems.