Language Modeling with Reduced Densities

TL;DR

This paper explores the mathematical structure of unstructured text data using enriched category theory, modeling sequences as enriched categories and mapping them to reduced density operators to capture probabilistic relationships.

Contribution

It introduces a novel categorical framework for text data, representing sequences as enriched categories and employing reduced density operators to model probabilistic structure.

Findings

Sequences form categories enriched over probabilities.

A functor maps text categories to reduced density operators.

The approach offers a new perspective on text entailment.

Abstract

This work originates from the observation that today's state-of-the-art statistical language models are impressive not only for their performance, but also - and quite crucially - because they are built entirely from correlations in unstructured text data. The latter observation prompts a fundamental question that lies at the heart of this paper: What mathematical structure exists in unstructured text data? We put forth enriched category theory as a natural answer. We show that sequences of symbols from a finite alphabet, such as those found in a corpus of text, form a category enriched over probabilities. We then address a second fundamental question: How can this information be stored and modeled in a way that preserves the categorical structure? We answer this by constructing a functor from our enriched category of text to a particular enriched category of reduced density operators. The latter leverages the Loewner order on positive semidefinite operators, which can further be interpreted as a toy example of entailment.