Gaussian mixture models as a proxy for interacting language models

TL;DR

This paper introduces an interacting Gaussian mixture model system as a computationally efficient proxy for large language models, capturing key interactive behaviors with theoretical analysis.

Contribution

It presents a novel GMM-based model mimicking LLM interactions, including RAG-like updates, with formal analysis of polarization phenomena.

Findings

Interacting GMMs can simulate aspects of LLM interactions.

The model enables analysis of polarization in LLM-like systems.

Lower bounds on polarization probability are established.

Abstract

Large language models (LLMs) are powerful tools that, in a number of settings, overlap with the results of human pattern recognition and reasoning. Retrieval-augmented generation (RAG) further allows LLMs to produce tailored output depending on the contents of their RAG databases. However, LLMs depend on complex, computationally expensive algorithms. In this paper, we introduce interacting Gaussian mixture models (GMMs) as a proxy for interacting LLMs. We construct a model of interacting GMMs, complete with an analogue to RAG updating, under which GMMs can generate, exchange, and update data and parameters. We show that this interacting system of Gaussian mixture models, which can be implemented at minimal computational cost, mimics certain aspects of experimental simulations of interacting LLMs whose iterative responses depend on feedback from other LLMs. We build a Markov chain from this system of interacting GMMs; formalize and interpret the notion of polarization for such a chain; and prove lower bounds on the probability of polarization. This provides theoretical insight into the use of interacting Gaussian mixture models as a computationally efficient proxy for interacting large language models.