Information Theory of Meaningful Communication

TL;DR

This paper proposes a novel approach to quantify the information conveyed in meaningful language by leveraging large language models to measure bits of meaning per clause, focusing on communication's semantic aspect.

Contribution

It introduces a method to measure meaningful communication in bits of meaning per clause using large language models, emphasizing semantic content over surface form.

Findings

Language differs from printed text in information units and focus on meaning.

Large language models can quantify bits of meaning in narratives.

The approach captures the informational content of communication beyond traditional entropy measures.

Abstract

In Shannon's seminal paper, entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its printed form: (i) the units of information are not characters or even words but clauses, i.e. shortest meaningful parts of speech; and (ii) what is transmitted is principally the meaning of what is being said or written, while the precise phrasing that was used to communicate the meaning is typically ignored. In this study, we show that one can leverage recently developed large language models to quantify information communicated in meaningful narratives in terms of bits of meaning per clause.