Emergence of order in random languages

TL;DR

This paper investigates the behavior of large texts generated by weighted context-free grammars, revealing phase transitions and the necessity of replica symmetry breaking in the information-carrying phase.

Contribution

It introduces a saddle-point analysis for large texts and demonstrates the need for replica symmetry breaking in the random language model.

Findings

Large texts are governed by saddle-point equations.

Replica symmetry breaking occurs in the information phase.

The model is solved in the high-temperature disordered phase.

Abstract

We consider languages generated by weighted context-free grammars. It is shown that the behaviour of large texts is controlled by saddle-point equations for an appropriate generating function. We then consider ensembles of grammars, in particular the Random Language Model of E. DeGiuli, Phys. Rev. Lett., 122, 128301, 2019. This model is solved in the replica-symmetric ansatz, which is valid in the high-temperature, disordered phase. It is shown that in the phase in which languages carry information, the replica symmetry must be broken.

Tables

Table 1. Table 1: Table of symbols. ‘msg’ = message. First block: basic symbols; second block: variables in diagrammatic representation; third block: variables in grammar ensemble.

Symbol	Definition	Linguistic Interpretation
$a,b,\dots$	alphabet of hidden symbols (non-terminal symbols)	abstract categories: noun, verb, noun phrase, …
$S$	start symbol	‘sentence’
$A,B,\dots$	alphabet of observable symbols	words: ‘the’, ‘bear’, ‘walked’, $\dots$
	(terminal symbols)
$M_{abc}$	weight for rule $a\to bc$	grammatical weight
$O_{aB}$	weight for rule $a\to B$	part-of-speech weight
${\sigma }_i$	hidden symbol on site $i$
$o_i$	observable symbol on site $i$
${\pi }_{abc}$	# of occurrences of rule $a\to bc$
${\rho }_{aB}$	# of occurrences of rule $a\to B$
$N$	# of hidden symbols	# of abstract categories
$T$	# of observable symbols	# of words in lexicon
$m$	# of sentences
$\ell$	# number of leaves	total number of words
${\ell }_0$	$\ell -m$
$L_a$	downward msg of left-child
$L_a^{†}$	upward msg of left-child
$R_a$	downward msg of right-child
$R_a^{†}$	upward msg of right-child
$H_a$	$L_a^{†}+R_a^{†}$	head message
$\eta ,\zeta ,\xi$	Lagrange multipliers to control
	# of branches, roots, leaves
${ϵ}_d$	deep temperature	inverse variance of $M$
${ϵ}_s$	surface temperature	inverse variance of $O$
${\widetilde{ϵ}}_d$	${ϵ}_d/N^3$
${\widetilde{ϵ}}_s$	${ϵ}_s/\left(NT\right)$

Equations

$$W(\{{\sigma }_i,o_t\}|\mathcal{T},\mathcal{G})=\prod _{\alpha \in {\Omega }_{\mathcal{T}}}{M}_{{\sigma }_{{\alpha }_1}{\sigma }_{{\alpha }_2}{\sigma }_{{\alpha }_3}}\prod _{\alpha \in \partial {\Omega }_{\mathcal{T}}}{O}_{{\sigma }_{{\alpha }_1}{o}_{{\alpha }_2}},$$

$$\mathbb{P}(\{{\sigma }_i,o_t\}|\mathcal{T},\mathcal{G})=\frac{W(\{{\sigma }_i,o_t\}|\mathcal{T},\mathcal{G})}{Z(\mathcal{T},\mathcal{G})}$$

$$Z(\mathcal{T},\mathcal{G})=\sum _{\{{\sigma }_i,o_t\}}W(\{{\sigma }_i,o_t\}|\mathcal{T},\mathcal{G})$$

$$\mathbb{Z}(\mathcal{G};\ell )=\sum _{\mathcal{T},\ell (\mathcal{T})=\ell }\mathbb{P}(\mathcal{T}|\mathcal{G})Z(\mathcal{T},\mathcal{G})$$

$$\mathbb{Z}(\mathcal{G};m,\ell )=\sum _{\{{\ell }_i\},{\sum }_{i=1}^m{\ell }_i=\ell }\prod _i\mathbb{Z}(\mathcal{G};{\ell }_i)$$

$$W(\{{\sigma }_i,o_t\}|\mathcal{T},\mathcal{G})=\prod _{a,b,c}{M}_{abc}^{{\pi }_{abc}}\prod _{a,B}{O}_{aB}^{{\rho }_{aB}}$$

$$E=-\sum _{a,b,c}{\pi }_{abc}\left[h_{abc}+\log M_{abc}\right]-\sum _{a,B}{\rho }_{aB}\left[k_{aB}+\log O_{aB}\right]$$

$$Z_{h,k}(\mathcal{T},\mathcal{G})=\sum _{\{{\sigma }_i,o_t\}}{e}^{-\beta E}$$

$$⟨E⟩=-\frac{\partial \log Z}{\partial \beta }$$

$$\displaystyle Q_{abc}(\mathcal{G})=\langle\delta_{\sigma_{\alpha_{1}},a}\big{(}N^{2}\delta_{\sigma_{\alpha_{2}},b}\delta_{\sigma_{\alpha_{3}},c}-1\big{)}\rangle=\frac{N^{2}}{\ell_{0}}\langle\pi_{abc}\rangle-\frac{1}{\ell_{0}}\sum_{b^{\prime},c^{\prime}}\langle\pi_{ab^{\prime}c^{\prime}}\rangle,$$

$$Q_2\equiv \overline{\sum _{a,b,c}{Q}_{abc}^2}=\frac{N^5(N^2-1)}{{\ell }_0^2}(q_0-q_1)$$

$$q_0=\overline{⟨{\pi }_{abc}{⟩}^2},q_1=\overline{⟨{\pi }_{abc}⟩⟨{\pi }_{a{b}^{\prime }{c}^{\prime }}⟩},$$

$$⟨{\pi }_{abc}⟩=\frac{1}{\beta }\frac{\partial \log Z}{\partial h_{abc}}$$

$$W=|G{|}^{-N/2}\int DU{e}^{-\frac{1}{2}{\sum }_{i,j}{U}_i{G}_{ij}^{-1}{U}_j}{e}^{\xi {\sum }_i{U}_i+\eta {\sum }_{i,j,k}{U}_i{U}_j{U}_k{M}_{ijk}},$$

$$\mathbb{F}(\mathcal{G})=\int DL\int DR{e}^{-\frac{1}{g}{\sum }_a\left[L_a{L}_a^{†}+R_a{R}_a^{†}\right]}{e}^I$$

$$I=\zeta h(L_1+R_1)+\xi \sum _a{O}_a(L_a^{†}+R_a^{†})+\eta \sum _{a,b,c}{M}_{abc}(L_a^{†}+R_a^{†})L_b{R}_c.$$

$$\mathbb{Z}(\mathcal{G};m,\ell )=m!{\oint }^{\prime }\frac{d\zeta }{{\zeta }^{1+m}}{\oint }^{\prime }\frac{d\xi }{{\xi }^{1+\ell }}{\oint }^{\prime }\frac{d\eta }{{\eta }^{1+\ell -m}}\mathbb{F}(\mathcal{G}),$$

$$h^m{g}^{2\ell -m}\prod _{a,b,c}{M}_{abc}^{{\pi }_{abc}}\prod _{aB}{O}_{aB}^{{\rho }_{aB}}$$

$$(2h{)}^m{g}^{2\ell -m}=p^{\ell }(1-p{)}^{2\ell -m}{Z}_{tree}^{-m}$$

$$L_a$$

$$L_a^{†}$$

$$R_a$$

$$R_a^{†}$$

$$\ell -m$$

$$\ell$$

$$m$$

$${\mathbb{P}}_G(M,O)$$

$$s_d=\frac{1}{N^3}\sum _{a,b,c}{\log }^2\left[\frac{M_{abc}}{\overline{M}}\right],s_s=\frac{1}{NT}\sum _{a,B}{\log }^2\left[\frac{O_{aB}}{\overline{O}}\right]$$

$$\overline{s_d}=(2{\widetilde{ϵ}}_d{)}^{-1},\overline{s_s}=(2{\widetilde{ϵ}}_s{)}^{-1}.$$

$$\overline{\log \mathbb{Z}(\mathcal{G})}=\overline{{\frac{\partial \mathbb{Z}(\mathcal{G}{)}^n}{\partial n}|}_{n=0}}={\frac{\partial }{\partial n}|}_{n=0}\overline{\mathbb{Z}(\mathcal{G}{)}^n},$$

$$A(\xi ,\eta ,L,R)$$

$$=\prod _{a,B}\overline{e^{\xi O_{aB}{\sum }_i{H}_a^i}}\prod _{a,b,c}\overline{e^{\eta M_{abc}{\sum }_i{H}_a^i{L}_b^i{R}_c^i}}$$

$$\sqrt{{\widetilde{ϵ}}_d/\pi }\int dm{e}^{-{\widetilde{ϵ}}_d{m}^2}{e}^{e^m\overline{M}x}$$

$$=\sum _{q\ge 0}\frac{{\overline{M}}^q{x}^q}{q!}{e}^{q^2/(4{\widetilde{ϵ}}_d)}$$

Full text

Keywords: context-free grammar, language, replicas

Note: The body is this work is as published in J. Phys. A: Math. Theor. 52 (2019) 504001. A Corrigendum has been added as Appendix B

Many complex systems have a generative, or linguistic, aspect. For example, protein structure is written in sequences of amino acids, a language of 20 different symbols. A large body of previous work has investigated the social aspect of linguistic systems, namely that different agents must find consensus regarding the meaning of symbols [2, 3, 4]. A complementary but necessary aspect of any linguistic system concerns the hidden structure within the sequences themselves, independent of communication. The most basic structural property is syntax: the rules that govern how symbols can be combined to create richer structures and thus carry information. In computer science and linguistics, generative grammar has proved to be a valuable formalism to describe syntax, in a generalized sense [5, 6, 7]. A generative grammar consists of an alphabet of hidden symbols, an alphabet of observable symbols, and a set of rules, which allow certain combinations of symbols to be replaced by others. From an initial start symbol S, one progressively applies the rules until only observable symbols remain; any sentence produced this way is said to be ‘grammatical,’ and the set of all such sentences is called the language of the grammar. The sequence of rule applications is called a derivation. The Chomsky hierarchy distinguishes grammars based on the complexity of the grammatical rules. In this work, we restrict our attention to context-free grammars (CFGs), for which derivations are trees (Figure 1).

There are many theoretical results on the capabilities of CFGs [7]. However, little is known about the statistical properties of large, typical grammars. Recently, there has been increasing interest in approaching the properties of syntax from the point of view of statistical [8, 1] and quantum [9, 10, 11] physics. In [1], a simple model of random languages was proposed, using weighted context-free grammars. (We refer the reader to this work for all necessary details about generative grammars). With numerical simulations it was shown that as a ‘temperature’ $\epsilon_{d}$ is lowered, the model has a transition from a ‘disordered’ phase in which the grammar just produces noise, to an ‘ordered’ phase in which the grammar produces sequences with a nontrivial structure. This transition is characterized by emergence of order in the hidden structure, as measured by an order parameter $Q_{2}$, defined below. As the temperature is lowered through $\epsilon_{*}$, $Q_{2}$ rapidly increases. With a simple scaling argument, the location of this transition was understood as the place where energetic fluctuations begin to be comparable to entropy. However, detailed information on the transition was not obtained in [1]. In this work we use the replica method, as well as diagrammatic methods, to study random languages. We will solve the Random Language Model of [1] in the replica-symmetric phase, and show that it quantitatively captures the behavior of $Q_{2}$, as shown in Figure 2. We will see that the replica symmetry must be broken in the ordered phase.

1 Partition functions

A CFG in Chomsky Normal form is defined by two types of rules: $a\to bc$, where one hidden symbol becomes two hidden symbols, and $a\to B$, where a hidden symbol becomes an observable symbol. In a weighted CFG (WCFG), we assign weights $M_{abc}$ and $O_{aB}$, respectively, to these rules. With these weights we build a weight for an entire derivation tree, $\mathcal{T}$, as follows. We write $\sigma$ for the hidden variables, and $o$ for the observables. These are indexed by their positions on the tree. We write $\Omega_{\mathcal{T}}$ for the set of internal factors, i.e. factors of the form $a\to bc$, and $\partial\Omega_{\mathcal{T}}$ for the boundary factors, i.e. those associated to $a\to B$ rules. The number of boundary factors is written $\ell_{\mathcal{T}}$, which is also the number of leaves. Since derivations are trees, the number of internal factors is $\ell_{\mathcal{T}}-1$. Write $\mathcal{G}$ for the objects $M$ and $O$, which specify the grammar. A table of commonly appearing symbols is shown in Table 1.

Consider a derivation tree, such as that depicted in Figure 1a. In a weighted context-free grammar, a tree $\mathcal{T}$ with hidden variables $\sigma_{i}$ and observables $o_{t}$ has a weight

[TABLE]

where each $\alpha=(\alpha_{1},\alpha_{2},\alpha_{3})$ is a factor in the order $\sigma_{\alpha_{1}}\to\sigma_{\alpha_{2}}\sigma_{\alpha_{3}}$. This defines a conditional probability measure on configurations

[TABLE]

where

[TABLE]

Eq.(2) specifies the probability of a derivation, given the tree. However, the topology of the tree is itself dynamical. We will consider that trees are chosen as a Bernoulli process: beginning from the root, each hidden variable either becomes an observable, with probability $p$, or branches into two hidden variables, with probability $1-p$. This implies that $\mathbb{P}(\mathcal{T}|\mathcal{G})=W_{tree}(\mathcal{T})/Z_{tree}$ with $W_{tree}(\mathcal{T})=p^{|\partial\Omega_{\mathcal{T}}|}(1-p)^{|\Omega_{\mathcal{T}}|}$. The ‘emission probability’ $p$ controls the size of trees 111$Z_{tree}=2p/(1+|2p-1|)=1$ for $p>1/2$.. The tree-averaged partition function is then

[TABLE]

as a function of the number of leaves $\ell$, and the partition function for $m$ sentences of total length $\ell$ is

[TABLE]

We have $\sum_{i}|\partial\Omega_{\mathcal{T}_{i}}|=\sum\ell_{i}=\ell$, and $\sum_{i}|\Omega_{\mathcal{T}_{i}}|=\sum(2\ell_{i}-1)=2\ell-m$, so that $\mathbb{Z}_{tree}\equiv\prod_{i}\mathbb{P}(\mathcal{T}_{i}|\mathcal{G})=p^{\ell}(1-p)^{2\ell-m}Z_{tree}^{-m}$ just gives a trivial factor. For now we suppress dependence on $m$ and $\ell$. Note that $\mathbb{Z}(\mathcal{G};m,\ell)$ is the weight for the grand canonical partition function $\sum_{m,\ell\geq 0}\mathbb{Z}(\mathcal{G};m,\ell)$; we will, however, work at fixed $m$ and $\ell$.

2 Energy, Entropy, and Order Parameters

It is convenient to add some auxiliary parameters in order to extract additional observables. First, we note that (1) can be written as

[TABLE]

where $\pi_{abc}(\sigma)$ is the usage frequency of rule $a\to bc$ and $\rho_{aB}(\sigma,o)$ is the usage frequency of $a\to B$. Adding external fields $h_{abc}$ and $k_{aB}$ let us define the energy of a configuration as

[TABLE]

Then we can generalize (3) to

[TABLE]

where we added a bias $\beta$. The original ensemble is recovered for $\beta=1$ and $h=k=0$. We see that the average energy is

[TABLE]

and it is natural to define the entropy of the grammar as $S=\beta\langle E\rangle+\log Z$.

In [1] it was argued that a natural order parameter for WCFGs is one that measures the extent to which rules are applied uniformly: if all rules $a\to bc$ and $a\to B$ have the same weight, the grammar carries no information, and sentences will be indistinguishable from noise. To measure order in the deep grammar, define first

[TABLE]

averaged over all interior vertices $\alpha$, and averaged over derivations. The normalization $\ell_{0}=\ell-m$ for $\mathbb{Z}$. A spin-glass order parameter specific to deep structure is

[TABLE]

where

[TABLE]

with $b^{\prime}\neq b,c^{\prime}\neq c$. Here we used the fact that when $h=0,k=0$, the permutation symmetry is restored upon disorder averaging. We see that

[TABLE]

so $q_{0}$ and $q_{1}$ can be obtained from derivatives of $\overline{\log Z}$ with respect to the field.

3 Diagrammatic formulation

We expect that universal properties of weighted context-free languages are contained in the behavior of $\overline{\log\mathbb{Z}}$ when $\ell$ becomes large, where $\overline{\;\cdot\;}$ is an average over grammars. In order to compute this object, we find it convenient to move to an alternative, particle, representation. In particular, we seek a model whose diagrammatic expansion gives the derivation trees we seek to count, with the appropriate weights. This technique has been widely used in the study of 2D gravity [13], and facilitated Kazakov’s solution of the Ising model on random surfaces [14, 15]. Later, it was shown in a simpler setting that this technique could be used to easily obtain results for spin models on random graphs [16, 17].

We begin with a simplified model. Consider the formal integral

[TABLE]

where the measure is normalized such that |G|^{-N/2}\int DU\exp(-\mbox{\frac{1}{2}}\sum_{i,j}U_{i}G^{-1}_{ij}U_{j})=1. Strictly speaking, the integral is only defined by its perturbative expansion in $\eta$; convergence requires that the real part of $G^{-1}$ be positive-definite. This expansion generates Feynman diagrams with cubic vertices. Each vertex gets a factor $\eta M_{ijk}$, and the expansion with respect to $\xi$ generates sources. By Wick’s theorem, each edge gets a factor $G_{ij}$, the propagator. The coefficient in this expansion of $\xi^{m}\eta^{k}$ thus counts all such diagrams, possibly disconnected, with $k$ vertices and $m$ sources, times an inverse symmetry factor [18] 222Such factors appear when diagrams, including all their colour indices, have nontrivial symmetries, like reflections. In the disordered case where $M_{abc}$ depends on all indices, these symmetry factors will not play a role since typical connected graphs will have no symmetries..

This is a skeleton of what we need to count derivation trees, but there are several elements missing: first, $W$ includes all graphs with cubic vertices, not only trees. Second, even if we could restrict the sum to trees, there is nothing in $W$ to distinguish leaves from roots, or to distinguish the left and right branches from a given hidden node.

One solution to these problems is to use matrices as the integration variables, because their diagrammatic expansion can be arranged to give planar diagrams in an appropriate large $N$ limit [19, 20]. However, for our problem this is overkill. Instead, we will consider a theory of complex scalar fields with colour indices, equivalent to a complex matrix model with matrices of size 1x1. We will have two scalar fields $L_{a}$ and $R_{a}$, with colour indices $a=1\ldots N$. Consider

[TABLE]

where † denotes complex conjugate and

[TABLE]

with $O_{a}=\sum_{B}O_{aB}$. The measure $DL=\prod_{a}dRe[L_{a}]dIm[L_{a}]/(\pi g)$ is normalized such that $\int DL\;e^{-\frac{1}{g}\sum_{a}L_{a}L_{a}^{\dagger}}=1$, and similarly for $R$.

The propagator is diagonal in the colour indices $a=1\ldots N$ and for each $a$ is such that $\langle L_{a}^{\dagger}L_{a}\rangle=g$; that is, the Feynman rules are as shown in Figure 3. The diagram corresponding to Figure 1a is shown in Figure 4. We claim that, apart from accidental symmetry factors,

[TABLE]

where $\oint^{\prime}=\oint/(2\pi i)$. The proof is as follows.

The perturbative expansion with respect to $\eta$ generates cubic vertices with distinguished heads and $L$ and $R$ ends; the vertices can be placed on the plane such that that their heads point up. The expansion and contour integral for $\xi$ generate $\ell$ sources of $L^{\dagger}$ and $R^{\dagger}$, which are the leaves. The expansion and contour integral for $\zeta$ generate $m$ sources of $L_{1}$ and $R_{1}$, which are the roots. An $L_{a}$ can only be connected to a $L_{a}^{\dagger}$, and a $R_{a}$ can only be connected to a $R_{a}^{\dagger}$. We can orient all edges from $L_{a}\to L_{a}^{\dagger}$ and $R_{a}\to R_{a}^{\dagger}$, and similarly for the half-edges in the cubic vertices, flowing from head to L and R branches. Then, from each root, we can define paths by following arrows; any path we take will go through some number of cubic vertices and end in a leaf. Therefore there are $m$ connected components, one for each root. Considered as a graph, we can count the number of edges as follows: each source generates half an edge, and each cubic vertex generates $3/2$ edges. The total number of edges is \mbox{\frac{1}{2}}(m+3(\ell-m)+\ell)=2\ell-m. The difference between the number of vertices and the number of edges is $(m+\ell+\ell-m)-(2\ell-m)=m$, which is the number of connected components. Therefore the graph is a forest.

We thus generate a forest of $m$ planar, rooted, trees, with $\ell$ leaves in total. The weight of each diagram is

[TABLE]

where $\pi_{abc}$ is the usage frequency of rule $a\to bc$ and $\rho_{aB}$ is the usage frequency of $a\to B$. This expansion counts diagrams with a degeneracy of $2^{m}$ since each tree root can be either an $L$ or an $R$. In the expansion, the different connected components are not ordered. We would like to distinguish forests by the order of the trees, so we multiply the result by $m!$. Choosing $g$ and $h$ such that

[TABLE]

for all $\ell$ and $m$ we have our result.

The virtue of working with (17) is that when $\ell\to\infty,m/\ell=\alpha=$ constant, the leading behavior can be extracted by a saddle-point analysis [21] 333This can be seen explicitly by considering the rescaling $L=\ell^{1/2}L^{\prime},R=\ell^{1/2}R^{\prime},\eta=\ell^{-1/2}\eta^{\prime},\xi=\ell^{1/2}\xi^{\prime},\zeta=\ell^{1/2}\zeta^{\prime}$. . There is one subtlety. The integration variables are the real and imaginary parts of $L$ and $R$, and the saddle-point equations should be taken with respect to these parts. The solutions to $Re[L_{a}]$ and $Im[L_{a}]$, which may be complex, are then added to produce $L_{a}=Re[L_{a}]+iIm[L_{a}]$, and similarly for $R$. By linearity, this is equivalent to taking saddle-point equations with respect to $L_{a}$ and $L_{a}^{\dagger}$, and treating $L_{a}$ and $L_{a}^{\dagger}$ as independent. It is convenient to write $H_{a}=L_{a}^{\dagger}+R_{a}^{\dagger}$ for a head. The saddle-point equations are

[TABLE]

for all $a$.

(20),(21) and their pairs (22),(23) have an interpretation as recursion equations, which are equivalent to the saddle-point limit of Tutte recursion relations or loop equations in related contexts [22, 23]. They are also related to self-consistent equations derived for spin glasses on trees 444For example, $L_{a}$ is analagous to what is called $\phi(\{\sigma^{a}\})$ in [16] and $g_{n}(\{\sigma_{a}\})$ in [24]. In these works, $\phi$ and $g_{n}$ are functions of $n$ Ising variables, so that they can take $2n$ different values. Below, we will replicate the $L_{a}$ so that they take $nN$ different values; $N=2$ is the Ising case.. Indeed, any $L_{a}$ node can either propagate into $L_{a}^{\dagger}$ and become a leaf, with weight $g\xi O_{a}$, or propagate into $L_{a}^{\dagger}$ and become another branch, with weight $g\eta\sum_{b,c}M_{abc}L_{b}R_{c}$, including all possibilities. This gives (20). Similarly, any $L_{a}^{\dagger}$ node is either the child of a root, with weight $g\zeta h\delta_{b1}$, or the child of a branch, with weight $g\eta\sum_{a^{\prime},a,c}M_{a^{\prime}ac}(L_{a}^{\dagger}+R_{a}^{\dagger})R_{c}$, including all possibilities. This gives (21).

For specific grammars, (20)-(26) can be explicitly analyzed. Indeed, after writing these equations in terms of real variables, these take the form of ‘context-free schema’ as defined in section VII.6.1 in [25]. This class of equations, mainly defined by the fact that coefficients in the right-hand side of (20)-(26) are positive, generically have square root singularities at their radius of convergence [25]. Moreover, they can be solved by iteration.

These equations are also related to cavity equations, or belief propagation equations, in the literature of disordered systems [26]. For any probabilistic model living on a tree, the cavity equations are a closed, self-consistent set of equations that can be used to compute the partition function. The variables of these equations, known as messages, are probability measures on the symbols of the graph. Here the symbols are the hidden symbols, hence each message can be considered a variable with a colour index $a=1\ldots N$. In general, there are two messages per site: one going into the interaction, which here is a branch, and one outgoing. For a WCFG, the interaction at a branch depends only on the values of the hidden symbols there; there is no intrinsic site-to-site disorder. Therefore for large trees, one can seek a solution to the cavity equations that is independent of site, only depending on whether sites are the left or right children of their parent. In this case, the cavity equations become (20)-(26), up to some normalization factors. Therefore the variables $L$ and $R$, which above were introduced as dummy variables to generate the diagrammatic expansion, in fact have their proper interpretation as messages: $L$ is a downward message, while $L^{\dagger}$ is an upward message. In an iterative scheme to solve the cavity equations, these variables have additional time indices, hence the dynamical aspect of their name; (20)-(26) are the fixed-point equations.

In what follows, our interest, however, is in extracting the behavior of typical grammars in the case when $N$ is large. For this we need to choose an ensemble.

4 Grammar ensembles

We will consider two models. In [1] it was argued that a generic model will have lognormally distributed weights, viz.,

[TABLE]

where the deep and surface sparsities $s_{d}$ and $s_{s}$ are defined by

[TABLE]

and $J=e^{-\sum_{a,b,c}\log M_{abc}-\sum_{a,B}\log O_{aB}}$. Here $\overline{M}=1/N^{2}$ and $\overline{O}=1/T$. A plot of the weights for a range of $\tilde{\epsilon}_{d}=\epsilon_{d}/N^{3}$ is shown in Fig.5a. It is straightforward to show that $\epsilon_{d}$ and $\epsilon_{s}$ satisfy

[TABLE]

where $\overline{\;\cdot\;}$ denotes a grammar average and $\tilde{\epsilon}_{s}=\epsilon_{s}/(NT)$. A small ‘deep temperature’ $\epsilon_{d}$ corresponds to a large deep sparsity. The model (27) was called in [1] the Random Language Model (RLM).

It was shown in [1] that the RLM shows two phases, depending on the value of $\tilde{\epsilon}_{d}$, plus logarithmic corrections. More precisely, Shannon entropies appear to collapse with respect to $\tilde{\epsilon}_{d}\log^{k}N$, where $k=1$ or $k=2$ depending on the quantity considered. For $\tilde{\epsilon}_{d}\log^{k}N\gtrsim 1$, Shannon entropies are independent of the deep temperature $\epsilon_{d}$, and take maximal values, indicating that the grammar does not carry information: despite strictly following the rules of a WCFG, sentences are indistinguishable from random noise. For smaller $\epsilon_{d}$, entropies drop, and the grammar carries nontrivial information. It is our goal to extract this transition from (17).

It will turn out to be much simpler to consider an alternative model, where the weights $M_{abc}$ and $O_{aB}$ are Gaussian, rather than lognormal; matching the mean and variance to those of the RLM we can again use the quantities $\overline{M}$ and $\tilde{\epsilon}_{d}$, and similarly for $O_{aB}$. The distribution is plotted in Fig.5b. This model has the unphysical feature that weights have a negative tail; naively, we could imagine that this would be unimportant, since the largest weights are most important, but we will have to revise this statement later. We call this the Gaussian model (GM).

We wish to compute

[TABLE]

where we used the replica method [27]. The fields $\zeta,\xi,\eta,L$, and $R$ are all replicated, adding an index $i=1\ldots n$. To compute $\overline{\mathbb{F}(\mathcal{G})^{n}}$ we need the grammar average

[TABLE]

Write $x_{abc}=\sum_{i}\eta_{i}H_{a}^{i}L^{i}_{b}R^{i}_{c}$. In the RLM the grammar averages over $M$ are of the form ($m=\log M_{abc}/\overline{M}$)

[TABLE]

A term of order $q$ corresponds to the rule appearing $q$ times. We are interested in a transition due to patterns of repeated rule application between sentences, rather than inside them (this would correspond to a transition deeper in the ordered phase). Therefore a priori we expect that we only need connected terms up to a small finite order $q_{*}$. Note that at order $q_{*}$, terms involving $q_{*}$ different replicas will be present. Resumming gives

[TABLE]

with a_{1}=e^{1/(4\tilde{\epsilon}_{d})},a_{2}=\mbox{\frac{1}{2}}(e^{4/(4\tilde{\epsilon}_{d})}-e^{2/(4\tilde{\epsilon}_{d})}), etc. From this divergent sum we only need to retain terms up to order $q=n(\ell-m)$, since the integration over $\eta$ will retain only $\ell-m$ vertices for each replica.

For practical reasons exact calculations are limited to $q_{*}=2$. In this case, we consider all derivations in which a rule can appear at most twice in one derivation tree. Note that rules can still appear arbitrarily many times in the set of $n$ replicas and $m$ sentences. Keeping terms to $q_{*}=2$ is equivalent to letting the $M$ be drawn from a Gaussian distribution. For appropriate choice of mean and variance, we can thus fix the GM to be equal to the RLM to this order. In the remainder of this work, we will first find the exact solution of the GM, and then discuss its extension to the RLM.

4.1 Gaussian model

Applying the same arguments to the integral over $O$, we have for the GM

[TABLE]

where we introduced ‘magnetization’ vectors and overlap matrices

[TABLE]

and b_{1}=e^{1/(4\tilde{\epsilon}_{s})},b_{2}=\mbox{\frac{1}{2}}(e^{4/(4\tilde{\epsilon}_{s})}-e^{2/(4\tilde{\epsilon}_{s})}). (Recall that $H_{a}=L_{a}^{\dagger}+R_{a}^{\dagger}$.). Assembling the above results we find that for the GM,

[TABLE]

This is now in the form amenable to standard treatment by replicas: the overlap matrices can be introduced as new parameters, and the original variables can be integrated out. We notice that the colour indices play the role usually played by spatial indices in spin glasses [27]. The surprising result is that the model can be exactly integrated, without even making an ansatz on the replica structure, and without taking the large $\ell$ limit. This integrability can be traced to the fact that the overlap matrices depend only on the real and imaginary parts of $L$ in the canonical way, i.e. through $L_{a}^{i}=Re[L_{a}^{i}]+iIm[L_{a}^{i}]$. This gives rise to a symplectic structure that simplifies the integration over these variables. The derivation is sketched in Appendix A. The final result is

[TABLE]

with $\ell_{0}=\ell-m$,

[TABLE]

and

[TABLE]

Here $x_{d}=N^{-3/2}\sqrt{e^{1/(2\tilde{\epsilon}_{d})}-1}$ and $x_{s}=(NT)^{-1/2}\sqrt{e^{1/(2\tilde{\epsilon}_{s})}-1}$.

The elements of $\overline{\log\mathbb{Z}}$ are as follows. Those involving $h$ and $g$ give $\log\mathbb{Z}_{tree}$, the contribution to $\log\mathbb{Z}$ that weights configurations by the number of sentences and leaves; these are precisely those required from 18. $S_{\ell,m}$ is entropic, and apparently counts the number of ways to partition a total string of length $\ell$ into $m$ sentences. Terms with $b_{j}$ and $a_{j}$ are energetic, since they depend on the grammar weight distribution. Those involving $b_{1}$ and $a_{1}$ capture the change in the mean occupancy as temperature is varied. The function $f(x)$ captures the non-trivial effects of correlation between different symbols. This function, which can written in terms of hypergeometric functions, is plotted in Fig.6. It develops an imaginary part for $x\sim\mathcal{O}(1)$, indicating that the unphysical negative probability states are becoming important. For large $N$ the condition $x_{d}\lesssim\mathcal{O}(1)$ is equivalent to

[TABLE]

and similarly the condition $x_{s}\lesssim\mathcal{O}(1)$ is equivalent to $\tilde{\epsilon}_{s}\log NT\gtrsim 1/2$. These inequalities fix the regime in which the GM is physical.

4.2 RLM

We now return to the full model. As discussed above, we cannot obtain the exact solution; however, since a saddle-point method is justified for large $\ell$, we can consider different ansatze on the form of the solution. Two are natural: (i) the colour-symmetric ansatz $L_{a}^{i}=L^{i},R_{a}^{i}=R^{i}$ $\forall a$, and the (ii) replica-symmetric ansatz $L_{a}^{i}=L_{a},R_{a}^{i}=R_{a}$ $\forall i$. After some calculations similar to those for the GM, we eventually find for either (i) or (ii) the same form (38), except that $f\equiv 0$. Besides the ansatze on the form of $L$ and $R$, we assume that $\ell$ is large and that the replica limit $n\to 0$ can be taken perturbatively, i.e. keeping terms $\mathcal{O}(n)$ in the action, as in the usual approach [27] 555Consistency with the GM suggests that we should be able to recover (38) with $f\equiv 0$ for that model, without necessarily taking the limit $\tilde{\epsilon}_{d}\to\infty,\tilde{\epsilon}_{s}\to\infty$, in which case the $f$ terms are trivially unimportant. Indeed, if instead of taking $n\to 0$ in (49), we look for a saddle-point with large $\ell$, the saddle-point perturbative in $n$ gives exactly (38) with $f\equiv 0$. This indicates that the function $f$ is nonperturbative in the replica limit.. We now analyze (38) with $f\equiv 0$, with the understanding that this holds in the replica-symmetric regime, whose range of validity is to be determined.

It is convenient to separate $F=-\overline{\log\mathbb{Z}}$ into its entropic and energetic contributions. This can be done exactly because the RLM has a scaling symmetry when the bias $\beta$ is included. Indeed, it is not hard to show that the partition function satisfies the scaling property $\overline{\mathbb{Z}^{n}}(\beta,\tilde{\epsilon}_{d},\overline{M},\tilde{\epsilon}_{s},\overline{O})=\overline{\mathbb{Z}^{n}}(1,\tilde{\epsilon}_{d}/\beta^{2},\overline{M}^{\beta},\tilde{\epsilon}_{s}/\beta^{2},\overline{O}^{\beta})$ (in abuse of earlier notation). The $\beta-$dependent part of $F$ is

[TABLE]

so that at $\beta=1$, $E=-\ell\log{\overline{O}}-\ell_{0}\log{\overline{M}}-\frac{\ell}{2\tilde{\epsilon}_{s}}-\frac{\ell_{0}}{2\tilde{\epsilon}_{d}}$ and the replica-symmetric entropy is

[TABLE]

The entropy cannot be negative; this gives necessary, though perhaps not sufficient, conditions on the regime where the replica-symmetric ansatz is applicable. For simplicity, consider the case $\ell\to\infty,m/\ell=\alpha\ll 1$ ($\ell/m$ is the typical length of a tree; we let it be large). Then one can determine that $S_{\ell,m}=\mathcal{O}(\alpha\ell)$ and, for the simulated case where the emission probability is close to $p=1/2$, $g=h\approx 1/\sqrt{8}$. The condition $S_{RS}>0$ is approximately equivalent to

[TABLE]

Our main concern is the emergence of deep structure, which does not depend on what happens at the surface of the tree. In the limit $\tilde{\epsilon}_{s}\to\infty$, this becomes $\tilde{\epsilon}_{d}\log(N^{2}T/8)\gtrsim 1/4$, very similar to the regime in which the GM is physical, 41.

4.3 Order parameter

Finally, we return to the order parameter $Q_{2}$ that measures deep structure. Let us first give a heuristic derivation of its value in the RLM. We need to compute $q_{0}=\overline{\langle\pi_{abc}\rangle^{2}}$ and $q_{1}=\overline{\langle\pi_{abc}\rangle\langle\pi_{ab^{\prime}c^{\prime}}\rangle}$, where $\pi_{abc}$ is the occupancy of the rule $a\to bc$. Using (7) and (13) in the diagrammatic representation, one can see that $\pi_{abc}=e^{h_{abc}}\eta M_{abc}H_{a}L_{b}R_{c}$. The $\pi_{abc}$ satisfy the sum rule $\sum_{abc}\pi_{abc}=\ell_{0}$ and so have a mean value $\overline{\pi}=\ell_{0}/N^{3}$. The occupancies are positively correlated with the grammar weights, since rules with higher weights are sampled more frequently. A crude estimate is then $\langle\pi_{abc}\rangle/\overline{\pi}\sim M_{abc}/({\overline{M}}a_{1})$ (the mean value of a weight is ${\overline{M}}a_{1}$). This leads to the estimate

[TABLE]

which indicates that order increases as $\epsilon_{d}$ is lowered simply because the weight variance increases. $Q_{2}$ can be computed more precisely using replicas. After a long computation, assuming the replica symmetric ansatz and taking large $\ell$, one eventually finds exactly the same result, (45). Thus this simple expression is in fact the genuine replica symmetric result; it is plotted in Fig.2, where it is compared with numerical data 666These data have been obtained by the same methods as described in [1]. Here we have simulated many more samples ($\ell\sim 10^{5}$ compared to $\ell\sim 10^{3}$ in that work) in order to resolve the large $\epsilon_{d}$ part of the curve.. In the large $\ell$ limit, it matches quantitatively, without fitting parameters, above $\tilde{\epsilon}_{d}\log N\gtrsim 1$. For smaller $\tilde{\epsilon}_{d}$, the data asymptote, as they must, while the replica-symmetric prediction diverges.

5 Conclusion

We showed that the partition function for weighted context-free grammars has a convenient diagrammatic representation. For individual grammars, the behavior of a large text $\ell\gg 1$ is governed by saddle-point equations, which resemble belief-propagation equations [26].

We then considered two ensembles of grammars, which are equivalent in a large temperature limit. The Gaussian model (GM) was solved exactly, and shown to become unphysical for $\tilde{\epsilon}_{d}\log N\lesssim 1/6$. For the random language model (RLM), previously simulated in [1], the partition function was computed in the replica-symmetric ansatz; the entropy becomes negative at low temperature, again depending essentially on the quantity $\tilde{\epsilon}_{d}\log N$. Finally, the order parameter $Q_{2}$ was computed in the replica-symmetric ansatz. The prediction quantitatively agrees with simulations above $\tilde{\epsilon}_{d}\log N\gtrsim 1$. These results indicate that replica-symmetry must be broken in the nontrivial low-temperature phase.

The RLM bears some similarity to a spin-glass on the Bethe lattice, a difficult problem that is still not fully understood [28, 29]. Indeed, both problems can be generated by a diagrammatic method [17], and in both problems one finds that overlaps of all orders are needed to compute the partition function [30, 28, 24, 29]. However, for the spin-glass, one can perform an expansion around the mean-field limit [30, 28, 24], which is the Sherrington-Kirkpatrick model solved by Parisi [27]. Naïvely, the analogue to the SK model would be the Gaussian model, which we solved above. However, we showed that this model does not break the replica symmetry. This is related to a gauge symmetry in the diagrammatic formulation. It is therefore an open question whether there is a more primitive model that captures the essence of random languages in the low-temperature phase, and remains solvable.

Finally, we have focussed here on context-free grammars, for which derivations are trees. The next level up in the Chomsky hierarchy are context-sensitive grammars. A theorem of Kuroda [31] says that it is sufficient to add rules of the form $ab\to cd$ to those above to model all context-sensitive grammars. Clearly this will add a quartic vertex to our (16), which is not in itself a difficulty. However, well-formed derivations must be represented by planar diagrams, so that the order of symbols is preserved in the derivation. Generating random planar graphs that are not trees requires matrices as integration variables; this strongly suggests that general grammars require the full machinery of complex matrix models.

I am grateful to V. Kazakov and G. Semerjian for conversations at an early stage of this work, and to G. Parisi and W. Bialek for encouragement.

Appendix A Solution of Gaussian model

We introduce $Q_{L}^{ij}=\sum_{a}L_{a}^{i}L_{a}^{j}$ as a new variable with a corresponding momentum $\Lambda_{L}^{ij}$, and similarly for $Q_{R}^{ij}=\sum_{a}R_{a}^{i}R_{a}^{j}$ and $P^{ij}=\sum_{a}L_{a}^{i}R_{a}^{j}$, with conjugate momenta $\Lambda_{R}$ and $\Lambda_{P}$, respectively. Let us write $x_{a}^{i}=Re[L_{a}^{i}],y_{a}^{i}=Im[L_{a}^{i}]$. The variables $\{x_{a}^{i},y_{a}^{i}\}$ are Gaussian, with a coupling matrix diagonal in colour. For each $a$, the coupling matrix is a $2n\times 2n$ matrix acting on $(x_{a}^{1},\ldots x_{a}^{n},y_{a}^{1},\ldots,y_{a}^{n})$,

[TABLE]

where ${\hat{\delta}}$ is the $n\times n$ identity matrix. It is easily verified that $Z$ is complex symplectic: $J=Z^{t}\cdot J\cdot Z$, where

[TABLE]

This implies that $Z^{-1}=J^{-1}\cdot Z^{t}\cdot J$ and, less obviously, $|Z|=1$ [32]. Hence after integrating out $\{L_{a}^{i}\}$ there is no nontrivial entropic term from $\log|Z|$, nor does there appear the inverse of $\Lambda_{L}$, as would naively be expected. In fact, after integrating out $\{L_{a}^{i}\}$ and $\{R_{a}^{i}\}$ the action remains linear in $\Lambda_{L}$, $\Lambda_{R}$, and $\Lambda_{P}$. Hence these can be immediately integrated out and we find that

[TABLE]

We find

[TABLE]

with $\tilde{b}_{j}=NT{\overline{O}}^{j}b_{j},\tilde{a}_{j}=N^{3}{\overline{M}}^{j}a_{j}$, $j=1,2$. The quadratic terms can be linearized with a Hubbard-Stratonovich transformation, after which the integrals over $\zeta,\xi$, and $\eta$ are simple. The result is

[TABLE]

with

[TABLE]

The calculation is thus reduced to an effective single-colour problem. We have

[TABLE]

so that the final result is

[TABLE]

with

[TABLE]

and $S_{\ell,m}=\log(\ell-m)!-\log\ell!+\log(1+2\ell-m)!-\log(1+2\ell-2m)!$. Letting $x_{d}=\sqrt{2\tilde{a}_{2}/\tilde{a}_{1}^{2}}=N^{-3/2}\sqrt{e^{1/(2\tilde{\epsilon}_{d})}-1}$ and $x_{s}=(NT)^{-1/2}\sqrt{e^{1/(2\tilde{\epsilon}_{s})}-1}$, we have the result shown in the main text.

Appendix B Corrigendum to J. Phys A: Math. Theor. 52 (2019) 504001

*The above preprint proposed a diagrammatic formulation of the Random Language Model; explained why the model is dominated by saddle-points; and sought the solution to the disorder-averaged model by comparison to a simpler, solvable model. We discuss a hidden assumption of the latter analysis above that was neither explained nor motivated: the analytical solution to the Gaussian model, and its extension to the Random Language Model, are predicated on a “downwards” approximation that neglects information flow from the leaves to the root of derivation trees. *

The diagrammatic formulation of the Random Language Model (RLM) [1] uses complex-valued fields $L_{a}^{i}$ and $R_{a}^{i}$ where $a=1\ldots N$ is a colour index and $i=1\ldots n$ is a replica index. Above we proposed a replica-symmetric solution to this model, using first a Gaussian model (GM) where the analysis is simplified. The solution to the GM was sketched in the Appendix. Here we point out a hidden assumption of the analysis above.

Write $L_{a}^{j}=x_{a}^{j}+iy_{a}^{j}$ and $R_{a}^{j}=z_{a}^{j}+iw_{a}^{j}$. After taking the disorder average, the GM depends nonlinearly on several order parameters

[TABLE]

The strategy for integrating the model is to introduce these order parameters as new integration variables. In so doing the original variables $L$ and $R$ are decoupled and can be integrated out. Since these are all complex-valued parameters, we need to treat their real and imaginary parts separately. For example, we can introduce $Q_{L}$ as $Q_{L,R}+iQ_{L,I}$ with

[TABLE]

$Q_{R}$, $L_{*}$, and $R_{*}$ can be introduced in a similar manner. We can write $Q_{H}={Q^{\dagger}}_{L}+{Q^{\dagger}}_{R}+{P^{\dagger}}+{P^{\dagger}}^{T}$ with

[TABLE]

and then introduce $P$ as a new variable, again in real and imaginary parts. After introducing these quantities the partition function takes the form

[TABLE]

where $D[...]$ includes all integrations, including those over $\zeta$, $\xi$, and $\eta$ (see above). The action $S_{0}$ contains the nontrivial RLM-specific dependence on the overlaps and ‘magnetization’ vectors. The action $S_{1}$ contains the inner products of the overlap matrices and magnetization vectors with their corresponding Lagrange multipliers:

[TABLE]

Finally the action $S_{2}$ includes the integrals over the $\{x_{a}^{i},y_{a}^{i},z_{a}^{i},w_{a}^{i}\}$, which can be done explicitly. It is a function of all the Lagrange multipliers. The remaining integrals in $\overline{Z}$ are determined by saddle-point equations.

To simplify the problem, one can make an ansatz to reduce the problem complexity (for example that overlap matrices are of hierarchical (Parisi) type). Alternatively, here we have real and imaginary parts of variables and can make an ansatz on this structure.

To motivate approximations, we recall from above that the diagrammatic formulation is related to the cavity method of disordered systems [27]. $L$ has the interpretation of a downward message (from the root to the leaves), while ${L^{\dagger}}$ has the interpretation of an upward message (from the leaves to the root). Therefore variables in the action can be classified by whether they propagate information upwards (made from ${L^{\dagger}}$ and/or ${R^{\dagger}}$) or downwards (made from $L$ and/or $R$). Because of the model structure, it is natural that downwards messages are more important for observables that depend on the bulk variables (as opposed to the surface, which is formed by the leaves of the tree). Indeed the probability of any bulk observable can be written as a nontrivial function of the $M$ grammar only, multiplied by the normalization constant $1/Z$. The surface properties only affect probabilities through the normalization $Z$.

Suppose, for the sake of argument, that we neglect the upwards information transfer. This is equivalent to neglecting the ${Q^{\dagger}}_{L}$, ${Q^{\dagger}}_{R}$, and ${P^{\dagger}}$ dependence in $S_{0}$, but leaving the $Q_{L}$, $Q_{R}$, and $P$ dependence. Consider then the saddle-point equation

[TABLE]

Independence of $S_{0}$ with respect to ${Q^{\dagger}}$ implies $\frac{\partial S_{0}}{\partial Q_{L,R}^{ij}}+i\frac{\partial S_{0}}{\partial Q_{L,I}^{ij}}=0$ so that

[TABLE]

or $\Lambda_{L}=0$, schematically. Note that $\Lambda_{L}^{\dagger}=2\Lambda_{L,R}\neq 0$.

Applying this condition $\Lambda_{L}^{\dagger}$ appears in the action as

[TABLE]

which is the form that was used above.

As shown above, the resulting theory correctly predicts the behaviour of the order parameter $Q_{2}$ in the high-temperature regime of the model. This suggests a posteriori that the “downwards” approximation is justified in this regime. We expect that it breaks down in the nontrivial low-temperature regime.

This “downwards” approximation leads to a symplectic propagator. To see this, consider the inverse propagator for the $L$ variables, i.e.

[TABLE]

which now has a partially factorized colour, replica, and real/imaginary dependence. Define the replica part $\tilde{M}^{ij}$ by $M^{ij}_{ab}=(2/g)\delta_{ab}\tilde{M}^{ij}$, i.e.

[TABLE]

where $g^{\prime}=g/2$ and ${\hat{\delta}}$ is the $n\times n$ identity matrix. Let

[TABLE]

and note that

[TABLE]

which means that $\tilde{M}$ is complex symplectic. This implies that $\tilde{M}^{-1}=J^{-1}\cdot\tilde{M}^{T}\cdot J$ and also $\det\tilde{M}=1$. These relations simplify the analysis.

We note in passing that the “upwards” approximation that neglects downwards information flow would also lead to a (different) symplectic propagator.

The analysis of the GM (and by extension the RLM) without making this “downwards” approximation will be reported elsewhere.