Hyperbolic Deep Learning for Foundation Models: A Survey

TL;DR

This survey reviews how hyperbolic geometry can enhance foundation models by addressing limitations of Euclidean spaces, improving reasoning, generalization, and efficiency in large-scale AI systems.

Contribution

It provides a comprehensive overview of hyperbolic neural networks and their application to foundation models, highlighting recent advances and future research directions.

Findings

Hyperbolic spaces enable efficient hierarchical and power-law data embeddings.

Applying hyperbolic geometry improves reasoning and zero-shot generalization in foundation models.

Hyperbolic methods maintain parameter efficiency while enhancing model capabilities.

Abstract

Foundation models pre-trained on massive datasets, including large language models (LLMs), vision-language models (VLMs), and large multimodal models, have demonstrated remarkable success in diverse downstream tasks. However, recent studies have shown fundamental limitations of these models: (1) limited representational capacity, (2) lower adaptability, and (3) diminishing scalability. These shortcomings raise a critical question: is Euclidean geometry truly the optimal inductive bias for all foundation models, or could incorporating alternative geometric spaces enable models to better align with the intrinsic structure of real-world data and improve reasoning processes? Hyperbolic spaces, a class of non-Euclidean manifolds characterized by exponential volume growth with respect to distance, offer a mathematically grounded solution. These spaces enable low-distortion embeddings of hierarchical structures (e.g., trees, taxonomies) and power-law distributions with substantially fewer dimensions compared to Euclidean counterparts. Recent advances have leveraged these properties to enhance foundation models, including improving LLMs' complex reasoning ability, VLMs' zero-shot generalization, and cross-modal semantic alignment, while maintaining parameter efficiency. This paper provides a comprehensive review of hyperbolic neural networks and their recent development for foundation models. We further outline key challenges and research directions to advance the field.