On the Vocabulary of Grammar-Based Codes and the Logical Consistency of Texts

TL;DR

This paper introduces a new interpretation of Zipf-Mandelbrot's law based on information theory, involving grammar-based codes and properties of nonergodic processes, to relate text length, fact description, and vocabulary size.

Contribution

It develops a novel class of grammar-based codes and analyzes nonergodic processes to connect text structure with information-theoretic properties.

Findings

Texts describing n^β facts contain at least n^β / log n words.

Grammar-based codes effectively model word usage in texts.

Nonergodic processes exhibit predictable properties relevant to language modeling.

Abstract

The article presents a new interpretation for Zipf-Mandelbrot's law in natural language which rests on two areas of information theory. Firstly, we construct a new class of grammar-based codes and, secondly, we investigate properties of strongly nonergodic stationary processes. The motivation for the joint discussion is to prove a proposition with a simple informal statement: If a text of length $n$ describes $n^\beta$ independent facts in a repetitive way then the text contains at least $n^\beta/\log n$ different words, under suitable conditions on $n$. In the formal statement, two modeling postulates are adopted. Firstly, the words are understood as nonterminal symbols of the shortest grammar-based encoding of the text. Secondly, the text is assumed to be emitted by a finite-energy strongly nonergodic source whereas the facts are binary IID variables predictable in a shift-invariant way.