Functorial Language Models

TL;DR

This paper introduces functorial language models that leverage monoidal functors to compute probabilities over word sequences, enabling training of categorical compositional models directly from raw text data.

Contribution

It presents a novel functorial approach to language modeling, integrating category theory with distributional semantics for the first time.

Findings

Proof-of-concept implementation in DisCoPy

Demonstrates feasibility of functorial models on raw text

Provides a new framework for compositional distributional semantics

Abstract

We introduce functorial language models: a principled way to compute probability distributions over word sequences given a monoidal functor from grammar to meaning. This yields a method for training categorical compositional distributional (DisCoCat) models on raw text data. We provide a proof-of-concept implementation in DisCoPy, the Python toolbox for monoidal categories.