Recovering manifold structure in LLM responses through a joint Euclidean mirror

TL;DR

This paper introduces a geometric framework to analyze and compare large language model responses across different tuning parameters by embedding response distributions into a low-dimensional Euclidean space.

Contribution

It develops a mathematical method to represent response distribution dissimilarities in Euclidean space and proposes estimation procedures with proven asymptotic properties.

Findings

Response distributions vary along distinct directions in the embedding space.

The method enables estimation of tuning parameters from response samples.

Experimental results show the approach effectively captures model response geometry.

Abstract

Understanding the behavior of black-box large language models and determining effective means of comparing their performance is a key task in modern machine learning. We consider how large language models respond to a specific query by analyzing how the distributions of responses vary over different values of tuning parameters. We frame this problem in a general mathematical setting, treating the mapping from model parameters to response distributions as a structured family of probability measures, endowed with a geometry via a dissimilarity measure. We show how dissimilarities between response distributions can be represented in low-dimensional Euclidean space through a joint Euclidean mirror surface encoding the underlying geometry, which permits both qualitative and quantitative analysis of large language models and provides insight into predicting response distributions for different values of tuning parameters. We propose an estimation procedure for the underlying joint Euclidean mirror based on observed samples from the response distributions, and we prove its asymptotic properties. Additionally, we propose a statistically consistent procedure to infer the value of an unknown model parameter based on samples from the corresponding response distribution and the estimated joint Euclidean mirror. In an experimental setting with large language models, we find that changes in different tuning parameter values correspond to distinct directions in the embedding space, making it possible to estimate the tuning parameters that were used to generate a given response.