Entropy in Large Language Models

TL;DR

This paper analyzes the entropy of large language models (LLMs) as information sources and compares it to natural language, revealing that LLMs have lower word entropy than natural speech, with implications for training and information theory.

Contribution

It introduces a probabilistic model for LLM outputs and compares their entropy to natural language, providing a quantitative measure of information content in LLMs.

Findings

LLMs have lower word entropy than natural language.

The study models LLM outputs as stationary probabilistic sources.

Results inform understanding of information and uncertainty in LLM training.

Abstract

In this study, the output of large language models (LLM) is considered an information source generating an unlimited sequence of symbols drawn from a finite alphabet. Given the probabilistic nature of modern LLMs, we assume a probabilistic model for these LLMs, following a constant random distribution and the source itself thus being stationary. We compare this source entropy (per word) to that of natural language (written or spoken) as represented by the Open American National Corpus (OANC). Our results indicate that the word entropy of such LLMs is lower than the word entropy of natural speech both in written or spoken form. The long-term goal of such studies is to formalize the intuitions of information and uncertainty in large language training to assess the impact of training an LLM from LLM generated training data. This refers to texts from the world wide web in particular.