The magnitude of categories of texts enriched by language models

TL;DR

This paper introduces a novel framework for analyzing texts enriched by language models using category theory and metric space concepts, providing new insights into the probabilistic structure and complexity of generated texts.

Contribution

It explicitly defines text categories using next-token probabilities, computes the magnitude of associated metric spaces, and links these to entropy measures and homological invariants.

Findings

Magnitude function sums over prompts of Tsallis entropies and output cardinality

Derivative of magnitude function recovers Shannon entropies

Magnitude homology groups are explicitly described

Abstract

The purpose of this article is twofold. Firstly, we use the next-token probabilities given by a language model to explicitly define a category of texts in natural language enriched over the unit interval, in the sense of Bradley, Terilla, and Vlassopoulos. We consider explicitly the terminating conditions for text generation and determine when the enrichment itself can be interpreted as a probability over texts. Secondly, we compute the M\"obius function and the magnitude of an associated generalized metric space of texts. The magnitude function of that space is a sum over texts (prompts) of the $t$-logarithmic (Tsallis) entropies of the next-token probability distributions associated with each prompt, plus the cardinality of the model's possible outputs. A suitable evaluation of the magnitude function's derivative recovers a sum of Shannon entropies, which justifies seeing magnitude as a partition function. Following Leinster and Shulman, we also express the magnitude function of the generalized metric space as an Euler characteristic of magnitude homology and provide an explicit description of the zeroeth and first magnitude homology groups.