Uncovering Hierarchical Structure in LLM Embeddings with $\delta$-Hyperbolicity, Ultrametricity, and Neighbor Joining

TL;DR

This paper evaluates the geometric properties of large language model embeddings using $\,delta$-hyperbolicity, ultrametricity, and Neighbor Joining, revealing their hierarchical structure and correlation with task performance.

Contribution

It introduces a novel framework for analyzing LLM embeddings' geometric properties, highlighting their hierarchical and tree-like structures using three complementary metrics.

Findings

LLM embeddings show varying degrees of hyperbolicity and ultrametricity.

The geometric properties correlate with model performance.

Embeddings often exhibit hierarchical, tree-like organization.

Abstract

The rapid advancement of large language models (LLMs) has enabled significant strides in various fields. This paper introduces a novel approach to evaluate the effectiveness of LLM embeddings in the context of inherent geometric properties. We investigate the structural properties of these embeddings through three complementary metrics $\delta$-hyperbolicity, Ultrametricity, and Neighbor Joining. $\delta$-hyperbolicity, a measure derived from geometric group theory, quantifies how much a metric space deviates from being a tree-like structure. In contrast, ultrametricity characterizes strictly hierarchical structures where distances obey a strong triangle inequality. While Neighbor Joining quantifies how tree-like the distance relationships are, it does so specifically with respect to the tree reconstructed by the Neighbor Joining algorithm. By analyzing the embeddings generated by LLMs using these metrics, we uncover to what extent the embedding space reflects an underlying hierarchical or tree-like organization. Our findings reveal that LLM embeddings exhibit varying degrees of hyperbolicity and ultrametricity, which correlate with their performance in the underlying machine learning tasks.