Random Language Model

TL;DR

This paper explores a model of random languages generated by weighted context-free grammars, revealing a phase transition from noise to structured language as grammar weight distribution broadens, indicating the emergence of deep linguistic structure.

Contribution

It introduces a phase transition framework for understanding how structured language emerges from random generative models based on weighted grammars.

Findings

Identifies a transition from noise to structured language as weights vary

Demonstrates the emergence of deep structure in the language model

Analyzes the balance between energy and entropy in language formation

Abstract

Many complex generative systems use languages to create structured objects. We consider a model of random languages, defined by weighted context-free grammars. As the distribution of grammar weights broadens, a transition is found from a random phase, in which sentences are indistinguishable from noise, to an organized phase in which nontrivial information is carried. This marks the emergence of deep structure in the language, and can be understood by a competition between energy and entropy.