A Rose by Any Other Name Would Smell as Sweet: Categorical Homotopy Theory for Large Language Models

TL;DR

This paper introduces a categorical homotopy framework for large language models to better understand and handle the equivalence of different linguistic expressions, leveraging advanced mathematical concepts.

Contribution

It develops a novel categorical homotopy approach using Markov categories to model and analyze language equivalences in LLMs, addressing fundamental rephrasing issues.

Findings

Introduces LLM Markov category for language probability modeling

Applies categorical homotopy to capture weak equivalences in LLMs

Connects LLM analysis with higher algebraic K-theory and model categories

Abstract

Natural language is replete with superficially different statements, such as ``Charles Darwin wrote" and ``Charles Darwin is the author of", which carry the same meaning. Large language models (LLMs) should generate the same next-token probabilities in such cases, but usually do not. Empirical workarounds have been explored, such as using k-NN estimates of sentence similarity to produce smoothed estimates. In this paper, we tackle this problem more abstractly, introducing a categorical homotopy framework for LLMs. We introduce an LLM Markov category to represent probability distributions in language generated by an LLM, where the probability of a sentence, such as ``Charles Darwin wrote" is defined by an arrow in a Markov category. However, this approach runs into difficulties as language is full of equivalent rephrases, and each generates a non-isomorphic arrow in the LLM Markov category. To address this fundamental problem, we use categorical homotopy techniques to capture ``weak equivalences" in an LLM Markov category. We present a detailed overview of application of categorical homotopy to LLMs, from higher algebraic K-theory to model categories, building on powerful theoretical results developed over the past half a century.