A mathematical model for universal semantics

TL;DR

This paper introduces a universal, language-independent mathematical model that captures word meanings through numerical fingerprints, enabling cross-lingual understanding and semantic analysis without external resources.

Contribution

It presents a novel Markov process-based semantic model that derives low-dimensional, interpretable vectors representing concepts across multiple languages.

Findings

Effective extraction of topics and synonyms from texts.

Successful cross-language text matching and translation.

Quantification of word meanings across 14 languages.

Abstract

We characterize the meaning of words with language-independent numerical fingerprints, through a mathematical analysis of recurring patterns in texts. Approximating texts by Markov processes on a long-range time scale, we are able to extract topics, discover synonyms, and sketch semantic fields from a particular document of moderate length, without consulting external knowledge-base or thesaurus. Our Markov semantic model allows us to represent each topical concept by a low-dimensional vector, interpretable as algebraic invariants in succinct statistical operations on the document, targeting local environments of individual words. These language-independent semantic representations enable a robot reader to both understand short texts in a given language (automated question-answering) and match medium-length texts across different languages (automated word translation). Our semantic fingerprints quantify local meaning of words in 14 representative languages across 5 major language families, suggesting a universal and cost-effective mechanism by which human languages are processed at the semantic level. Our protocols and source codes are publicly available on https://github.com/yajun-zhou/linguae-naturalis-principia-mathematica

Equations

$$\left|{\delta }_i\right|<\frac{2}{\sqrt{n_{ii}}}\sqrt{\frac{{\pi }^2}{6}-1-\frac{1}{2n_{ii}}}$$

$$\mathbb{P}(\tau >t)\sim \sum _m{c}_m{e}^{-k_{m}t},$$

$$⟨\log L_{ii}⟩-\log ⟨L_{ii}⟩+{\gamma }_0$$

$$=$$

$$s_R({\mathbf{b}}_i^{\mathrm{A}},{\mathbf{b}}_j^{\mathrm{B}}):=\frac{\Vert {\mathbf{b}}_i^{\mathrm{A}}\wedge {\mathbf{b}}_j^{\mathrm{B}}{\Vert }_1}{\Vert {\mathbf{b}}_i^{\mathrm{A}}\vee {\mathbf{b}}_j^{\mathrm{B}}{\Vert }_1}$$

$$p_{ij}:=\frac{n_{ij}{e}^{-⟨\log L_{ij}⟩}}{\sum _{k=1}^N{n}_{ik}{e}^{-⟨\log L_{ik}⟩}},$$

$${\mathbf{P}}_{\mathrm{A}}{\mathbf{T}}_{\mathrm{A}\to \mathrm{B}}={\mathbf{T}}_{\mathrm{A}\to \mathrm{B}}{\mathbf{P}}_{\mathrm{B}}.$$

$$\mathbb{P}(⟨\log L_{ij}⟩>\ell )\approx {\alpha }_{ij}(\ell ):=\sqrt{\frac{n_{ij}}{2\pi {\beta }_i}}{\int }_{\ell }^{\infty }{e}^{-\frac{n_{ij}(x-{\ell }_i{)}^2}{2{\beta }_i}}\mathrm{d}x,$$

$${\ell }_i:=\frac{⟨L_{ii}\log L_{ii}⟩}{⟨L_{ii}⟩}-1,{\beta }_i:=\frac{⟨L_{ii}({\ell }_i-\log L_{ii}{)}^2⟩}{⟨L_{ii}⟩}.$$

$${\mathbf{v}}_i=(|{\lambda }_1({\mathbf{R}}^{[i]})|,|{\lambda }_2({\mathbf{R}}^{[i]})|,\dots ),$$

$$s_R({\mathbf{b}}_i^{\mathrm{A}},{\mathbf{b}}_j^{\mathrm{B}})\ge \max \left\{1-0.07\sqrt{K},1-\sqrt{\frac{\Vert {\mathbf{b}}_i^{\mathrm{A}}\wedge {\mathbf{b}}_j^{\mathrm{B}}{\Vert }_0}{\Vert {\mathbf{b}}_i^{\mathrm{A}}\vee {\mathbf{b}}_j^{\mathrm{B}}{\Vert }_1}}\right\}$$

$$\mathcal{F}[Q,A]:=-\sum _{i=1}^m\sum _{j=1}^n{\tilde{\pi }}_{Q_i}{\tilde{p}}_{Q_i{A}_j}\log {\tilde{p}}_{Q_i{A}_j}$$

$$Y_N:=\log \left(\frac{1}{N}\sum _{i=1}^N{X}_i\right)-\frac{1}{N}\sum _{i=1}^N\log X_i$$

$$\mathbb{P}\left(\frac{Y_N-\left({\gamma }_0-\frac{1}{2N}\right)}{\frac{1}{\sqrt{N}}\sqrt{\frac{{\pi }^2}{6}-1-\frac{1}{2N}}}<a\right)\sim {\int }_{-\infty }^a{e}^{-x^2/2}\frac{dx}{\sqrt{2\pi }}$$

$$\mathbb{E}{e}^{-t{Y}_N}$$

$$=$$

$$d{\xi }_1\cdots d{\xi }_N=r^{N-1}dr\prod _{j=1}^{N-1}{\sin }^{N-1-j}{\theta }_{j}d{\theta }_j.$$

$$\mathbb{E}{e}^{-t{Y}_N}=$$

$$=$$

$${\mathbb{E}}^{\mathsf{W}}(1_B\circ {\hat{\theta }}_n|{\mathcal{F}}_n)={\mathbb{E}}^{X(n)}(1_B):={\mathbb{P}}^{X(n)}(B),$$

$${\mathbb{E}}^{\mathsf{W}}(1_B\circ {\hat{\theta }}_n;A)={\mathbb{E}}^{\mathsf{W}}({\mathbb{E}}^{X(n)}(1_B);A).$$

$${\tau }_i(\omega ):=\inf \{n\in {\mathbb{Z}}_{>0}:X(n)={\mathsf{W}}_i\},$$

$${\tau }_{\mathcal{W}}(\omega ):=$$

$$=$$

$$H_{\mathcal{W}}(t):=\sum _{{\mathsf{W}}_i\in \mathcal{S}}{\mathbb{P}}^{{\mathsf{W}}_i}({\tau }_{\mathcal{W}}<t){\pi }_i,t\in {\mathbb{Z}}_{>0}.$$

$$R_{\mathcal{W}}(t):=\sum _{{\mathsf{W}}_i\in \mathcal{W}}{\mathbb{P}}^{{\mathsf{W}}_i}({\tau }_{\mathcal{W}}<t)\frac{{\pi }_i}{{\sum }_{{\mathsf{W}}_j\in \mathcal{W}}{\pi }_j},t\in {\mathbb{Z}}_{>0.}$$

$$H_{\mathcal{W}}(1)=R_{\mathcal{W}}(1)=0;\frac{H_{\mathcal{W}}(t+1)}{{\sum }_{{\mathsf{W}}_j\in \mathcal{W}}{\pi }_j}=\sum _{n=1}^t[1-R_{\mathcal{W}}(n)]$$

$$\sum _{{\mathsf{W}}_i\in \mathcal{S}}{\mathbb{P}}^{{\mathsf{W}}_i}({\tau }_{\mathcal{W}}\le t){\pi }_i=\sum _{n=1}^t\sum _{{\mathsf{W}}_j\in \mathcal{W}}[1-{\mathbb{P}}^{{\mathsf{W}}_j}({\tau }_{\mathcal{W}}<t)]{\pi }_j$$

$$\sum _{{\mathsf{W}}_i\in \mathcal{S}}{\mathbb{P}}^{{\mathsf{W}}_i}({\tau }_{\mathcal{W}}=1){\pi }_i=$$

$$=$$

$$\sum _{{\mathsf{W}}_i\in \mathcal{S}}{\mathbb{P}}^{{\mathsf{W}}_i}({\tau }_{\mathcal{W}}\le t+1){\pi }_i$$

$$=$$

Full text

1 Introduction

A quantitative model for the meaning of words not only helps us understand how we transmit information and absorb knowledge, but also provides foothold for algorithms in machine processing of natural language texts. Ideally, a universal mechanism of semantics should be based on numerical characteristics of human languages, transcending concrete written and spoken forms of verbal messages. In this work, we demonstrate, in both theory and practice, that the time structure of recurring language patterns is a good candidate for such a universal semantic mechanism. Through statistical analysis of recurrence times and hitting times, we numerically characterize connectivity and association of individual concepts, thereby devising language-independent semantic fingerprints (LISF).

Concretely speaking, we define semantics through algebraic invariants of a stochastic text model that approximately governs the empirical hopping rates on a web of word patterns. Such a stochastic model explains the distribution of recurrence times and outputs recurrence eigenvalues as semantic fingerprints. Statistics of recurrence times allow machines to tell non-topical words from topical ones. A comparison of hitting and recurrence times further generates quantitative fingerprints for topics, enabling machines to overcome language barriers in translation tasks and perform associative reasoning in comprehension tasks, like humans.

Akin to the physical world, there is a hierarchy of length scales in languages. On short scales such as syllables, words, and phrases, human languages do not exhibit a universal pattern related to semantics. Except for a few onomatopoeias, the sounds of words do not affect their meaning [1]. Neither do morphological parameters [2] (say, singular/plural, present/past) or syntactic rôles [3] (say, subject/object, active/passive). In short, there are no universal semantic mechanisms at the phonological, lexical or syntactical levels [4]. Grammatical “rules and principles” [2, 3], however typologically diverse, play no definitive rôle in determining the inherent meaning of a word.

Motivated by the observations above, we will build our quantitative semantic model on long-range and language-independent textual features. Specifically, we will measure the lengths of text fragments flanked by word patterns of interest (Fig. 1). Here, a word pattern is a collection of content words that are identical up to morphological parameters and syntactic rôles. A content word signifies definitive concepts (like apple, eat, red), instead of serving purely grammatical or logical functions (like but, of, the). Fragment length statistics will tell us how tightly/loosely one concept is connected to another. This in turn, will provide us with quantitative criteria for inclusion/exclusion of different concepts within the same (computationally constructed) semantic field. Such statistical semantic mining will then pave the way for machine comprehension and machine translation.

2 Methodology

We quantify the time structure of an individual word pattern $\mathsf{W}_{i}$ through the statistics of its recurrence times $\tau_{ii}$. We characterize the dynamic impact of a word pattern $\mathsf{W}_{i}$ on another word pattern $\mathsf{W}_{j}$ by the statistics of their hitting times $\tau_{ij}$. In what follows, we will describe the statistical analyses of $\tau_{ii}$ and $\tau_{ij}$, on which we build a language-independent Markov model for semantics.

2.1 Recurrence times and topicality

Assuming uniform reading speed,111On the scale of words (rather than phonemes), this assumption works fine in most languages that are written alphabetically. However, this working hypothesis does not extend to Japanese texts, which interlace Japanese syllabograms (lasting one mora per written unit) with Chinese ideograms (lasting one or more morae per written unit). we measure the recurrence times $\tau_{ii}$ for a word pattern $\mathsf{W}_{i}$ through $n_{ii}$ samples of the effective fragment lengths $L_{ii}$ (Figs. 1, 2a). Here, while counting as in Fig. 1, we ignore contacts between short-range neighbors, which may involve language-dependent redundancies.222For example, a German phrase *liebe Studentinnen und Studenten *with short-range recurrence is the gender-inclusive equivalent of the English expression dear students. Some Austronesian languages (such as Malay and Hawaiian) use reduplication for plurality or emphasis.

2.1.1 Recurrence of non-topical patterns

In a memoryless (hence banal) Poisson process (Fig. 2b), recurrence times are exponentially distributed (Fig. 2d,d*′*). The same is also true for word recurrence in a randomly reshuffled text [5]. If we have $n_{ii}$ independent samples of exponentially distributed random variables $L_{ii}$, then the statistic $\delta_{i}:=\log\langle L_{ii}\rangle-\langle\log L_{ii}\rangle-\gamma_{0}+\frac{1}{2n_{ii}}$ satisfies an inequality

[TABLE]

with probability 95% (see Theorem 1 in Appendix A for a two-sigma rule). Here, $\gamma_{0}:=\lim_{n\to\infty}\left(-\log n+\sum_{m=1}^{n}\frac{1}{m}\right)$ is the Euler–Mascheroni constant.

As a working definition, we consider a word pattern $\mathsf{W}_{i}$ non-topical if its $n_{ii}$ counts of effective fragment lengths $L_{ii}$ are exponentially distributed $\mathbb{P}(L_{ii}>t)\sim e^{-kt}$, within 95% margins of error [that is, satisfying (1) above].

2.1.2 Recurrence of topical patterns

In contrast, we consider a word pattern $\mathsf{W}_{i}$ topical if its diagonal statistics $n_{ii},L_{ii}$ constitute significant departure from the Poissonian line $\langle\log L_{ii}\rangle-\log\langle L_{ii}\rangle+\gamma_{0}=0$ (Fig. 2e, blue line), violating the bound in (1).

Notably, most data points for topics (colored dots on Fig. 2e) in Jane Austen’s Pride and Prejudice mark systematic downward departures from the Poissonian line. This suggests that the topical recurrence times $\tau=L_{ii}$ follow weighted mixtures of exponential distributions (Fig. 2c,c*′*):

[TABLE]

$\left(\text{where }c_{m},k_{m}>0,\text{ and }\sum_{m}c_{m}=1\right)$, which impose an inequality constraint on the recurrence time $\tau=L_{ii}$:

[TABLE]

2.1.3 Raw alignment of topical patterns

If a word pattern $\mathsf{W}_{i}$ qualifies as a topic by our definition (Fig. 3b,b*′*), then the signals in its coarse-grained timecourse (say, a vector $\mathbf{b}_{i}=(b_{i,1},\dots,b_{i,61})$ representing word counts in each chapter of Pride and Prejudice) are not overwhelmed by Poisson noise.

This vectorization scheme, together with the Ružička similarity [6]

[TABLE]

between two vectors with non-negative entries, allow us to align some topics found in parallel versions of the same document, in languages A and B (Fig. 3c). Here, in the definition of the Ružička similarity, $\wedge$ (resp. $\vee$) denotes component-wise minimum (resp. maximum) of vectors; $\|\mathbf{b}\|_{1}$ sums over all the components in $\mathbf{b}$.

2.2 Markov text model

2.2.1 Transition probabilities via pattern analysis

The diagonal statistics $n_{ii},L_{ii}$ (Fig. 1) have enabled us to extract topics automatically through recurrence time analysis (Figs. 2e and 3b,b*′*). The off-diagonal statistics $n_{ij},L_{ij}$ (Fig. 1) will allow us to determine how strongly one word pattern $\mathsf{W}_{i}$ binds to another word pattern $\mathsf{W}_{j}$, through hitting time analysis. In an empirical Markov matrix $\mathbf{P}=(p_{ij})$, the long-range transition rate $p_{ij}$ is estimated by

[TABLE]

where $n_{ij}$ counts the number of long-range transitions from $\mathsf{W}_{i}$ to $\mathsf{W}_{j}$, and $L_{ij}$ is a statistic that measures the effective fragment lengths of such transitions (Fig. 1).

2.2.2 Equilibrium state and detailed balance

Numerically, we find that our empirical Markov matrix $\mathbf{P}=(p_{ij})$ defined in (5) is a fair approximation to an ergodic333If a Markov chain is ergodic, then there is a strictly positive probability to transition from any Markov state (that is, any individual word pattern in our model) to any other state, after finitely many steps. matrix $\mathbf{P}^{*}=(p_{ij}^{*})$, which in turn, governs the stochastic hoppings between content word patterns during text generation.

Each ergodic Markov matrix $\mathbf{P}^{*}=(p_{ij}^{*})_{1\leq i,j\leq N}$ possesses a unique equilibrium state $\bm{\pi}^{*}=(\pi_{i}^{*})_{1\leq i\leq N}$. The equilibrium state $\bm{\pi}^{*}$ represents a probability distribution (that is, $\pi_{i}^{*}\geq 0$ for $1\leq i\leq N$ and $\sum_{i=1}^{N}\pi_{i}^{*}=1$) that satisfies $\bm{\pi}^{*}\mathbf{P}^{*}=\bm{\pi}^{*}$ (that is, $\sum_{1\leq i\leq N}\pi_{i}^{*}p_{ij}^{*}=\pi_{j}^{*}$ for $1\leq j\leq N$). In our numerical experiments, the dominant eigenvector $\bm{\pi}$ (satisfying $\bm{\pi}\mathbf{P}=\bm{\pi}$) consistently reproduces word frequency statistics that are proportional to the ideal equilibrium state $\bm{\pi}^{*}$ (Fig. 4a).

Furthermore, through numerical experimentation, we find that our empirical Markov matrix $\mathbf{P}\approx\mathbf{P}^{*}$ approximately honors the detailed balance condition $\pi_{i}^{*}p_{ij}^{*}=\pi_{j}^{*}p_{ji}^{*}$ for $1\leq i,j\leq N$. The approximation $\pi_{i}p_{ij}^{(n)}\approx\pi_{j}p_{ji}^{(n)}$ becomes closer as we go to higher iterates $\mathbf{P}^{n}=(p_{ij}^{(n)})$, where $n$ is a small positive integer (Fig. 4b).

On an ergodic Markov chain with detailed balance, one can show that recurrence times are distributed as weighted mixtures of exponential decays (see Theorem 3 in Appendix B.3), thus offering a theoretical explanation for (2).

2.2.3 Spectral invariance under translation

The spectrum $\sigma(\mathbf{P})$ (collection of eigenvalues) is approximately invariant against translations of texts (Fig. 4c), which can be explained by a matrix equation

[TABLE]

Here, both sides of the identity above quantify the transition probabilities from words in language A to words in language B, from the impressions of Alice and Bob, two monolingual readers in a thought experiment. On the left-hand side, Alice first processes the input in her native language A by a Markov matrix $\mathbf{P}_{\mathrm{A}}$, and then translates into language B, using a dictionary matrix $\mathbf{T}_{\mathrm{A}\to\mathrm{B}}$; on the right-hand side, Bob needs to first translate the input into language B, using the same dictionary $\mathbf{T}_{\mathrm{A}\to\mathrm{B}}$, before brainstorming in his own native language, using $\mathbf{P}_{\mathrm{B}}$. Putatively, the matrix equation holds because semantic content is shared by native speakers of different languages. In the ideal scenario where translation is lossless (with invertible $\mathbf{T}_{\mathrm{A}\to\mathrm{B}}$), the Markov matrices $\mathbf{P}_{\mathrm{A}}$ and $\mathbf{P}_{\mathrm{B}}$ are indeed linked to each other by a similarity transformation that leaves their spectrum intact.

2.3 Localized Markov matrices and semantic cliques

2.3.1 Semantic contexts for recurrent topics

Specializing spectral invariance to individual topical patterns, we will be able to generate semantic fingerprints through a list of topic-specific and language-independent eigenvalues. Here, we will be particularly interested in recurrence eigenvalues of individual topical patterns, which correspond to multiple decay rates in the weighted mixtures of exponential distributions.

Unlike the single exponential decays associated to non-topical recurrence patterns, the multiple exponential decay modes will enable our robot reader to easily discern one topic from another. In general, it is numerically challenging to recover multiple exponential decay modes from a limited amount of recurrence time measurements [7]. However, in text processing, we can circumvent such difficulties by off-diagonal statistics $n_{ij}$ and $L_{ij}$ that provide semantic contexts for individual topical patterns.

To quantitatively define the semantic content of a topical pattern $\mathsf{W}_{i}$, we specify a local, directed, and weighted graph, corresponding to a localized Markov transition matrix $\mathbf{P}^{[i]}$.

2.3.2 Localized Markov contexts of topical patterns

To localize, we need to remove edges between two vertices $\mathsf{W}_{i}$ and $\mathsf{W}_{j}$, when the hitting times $L_{ij}$ and $L_{ji}$ are “long enough” relative to what one could naïvely expect from recurrence time statistics $n_{ij},{n_{ji}}$ and $L_{ii},L_{jj}$. Here, for naïve expectation, we approximate the probability $\mathbb{P}(\langle\log L_{ij}\rangle>\ell)$ by a Gaussian model $\alpha_{ij}(\ell)$ (colored curves in Fig. 5a)

[TABLE]

whose mean and variance are deducible from $n_{ij}$ and $L_{ii}$ (see Theorem 4 in Appendix B.4):

[TABLE]

The parameters in the Gaussian model are justified by the relation between hitting and recurrence times [8] on an ergodic Markov chain with detailed balance, and become asymptotically exact if distinct word patterns are statistically independent (such as $\alpha_{13}$, $\alpha_{24}$, $\alpha_{31}$, $\alpha_{34}$ in Fig. 5a). Here, statistical independence justifies additivity of variances, hence the $\sqrt{n_{ij}}$ factor in (7); sums of independent samples of $\log L_{ij}$ become asymptotically Gaussian, thanks to the central limit theorem. Failing that, the actual ranking of $\langle\log L_{ij}\rangle$ may deviate from the Gaussian model prediction in (7), such as the intimately related pairs of words Elizabeth/Darcy, Elizabeth/Jane, Darcy/Elizabeth, Darcy/pride and pride/Darcy.

2.3.3 Markov criteria for semantic cliques

Empirically, we find that higher $\alpha_{ij}(\ell)$ scores point to closer affinities between word patterns (Fig. 5a), attributable to kinship (Elizabeth, Jane), courtship (Darcy, Elizabeth), disposition (Darcy, pride) and so on. Our robot reader automatically detects such affinities, without references other than the novel itself. Therefore, we can use the $\alpha_{ij}(\ell)$ scores as guides to numerical approximations of semantic fields, hereafter referred to as semantic cliques.

We invite a topical pattern $\mathsf{W}_{j}$ to the semantic clique $\mathscr{S}_{i}$ (Figs. 5a and b, insets) surrounding $\mathsf{W}_{i}$, if $\min\{\alpha_{ij}(\langle\log L_{ij}\rangle),\alpha_{ji}(\langle\log L_{ji}\rangle)\}>\alpha_{*}$ for a standard Gaussian threshold $\alpha_{*}:=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{1}e^{-x^{2}/2}\mathrm{d}\,x\approx 0.8413$. This operation emulates the brainstorming procedure of a human reader, who associates one word with another only when they stay much closer than two randomly picked words, according to his/her impression.

Indeed, by numerical brainstorming from $\mathsf{W}_{i}$, our semantic cliques $\mathscr{S}_{i}$ (Figs. 5a and b, insets) inform us about their center word $\mathsf{W}_{i}$, through several types of semantic relations, including, but not limited to

• •

Synonyms (pride and vanity in English, orgeuil and fierté in French, etc.);

• •

Temperaments (Elizabeth, a delightful girl, often laughs, corresponding to French verbs sourire and rire);

• •

Co-references (e.g. Darcy as a personification of pride);

• •

Causalities (such as pride based on fortune).

On a local graph with vertices $\mathscr{S}_{i}=\{\mathsf{W}_{i_{1}}=\mathsf{W}_{i},$ $\mathsf{W}_{i_{2}},$ $\dots,$ $\mathsf{W}_{i_{N_{i}}}\}$, we specify the connectivity of each directed edge by a localized Markov matrix $\mathbf{P}^{[i]}=(p_{jk}^{[i]})_{1\leq j,k\leq N_{i}}$. This localized Markov matrix is the row-wise normalization of an $N_{i}\times N_{i}$ subblock of $\mathbf{P}$ with the same set of vertices as $\mathscr{S}_{i}$. Resetting the entries $p_{1k}^{[i]}$ and $p^{[i]}_{j1}$ as zero, one arrives at the localized recurrence matrix $\mathbf{R}^{[i]}$. We call $\mathbf{R}^{[i]}$ a recurrence matrix, because one can use it to compute the distribution for recurrence times to the Markov state $\mathsf{W}_{i}$ in $\mathscr{S}_{i}$. As we will see soon in the applications below, the eigenvalues of $\mathbf{R}^{[i]}$, when properly arranged, become language-independent semantic fingerprints.

3 Applications

3.1 Automated word translations from bilingual documents

Experimentally, we resolve the connectivity of an individual pattern $\mathsf{W}_{i}$ through the recurrence spectrum $\sigma(\mathbf{R}^{[i]})$ (Fig. 5b). The dominant eigenvalues of $\mathbf{R}^{[i]}$ are concept-specific while remaining nearly language-independent (a localized version of the invariance in Fig. 4c). Such empirical evidence motivates us to define the language-independent semantic fingerprint (LISF) of a word pattern $\mathsf{W}_{i}$ by a descending list for the magnitudes of eigenvalues

[TABLE]

computable from its semantic clique $\mathscr{S}_{i}$. We zero-pad this vector from the $(\lfloor e^{\eta_{i}}\rfloor+1)$st component onwards, where $\eta_{i}$ is the Kolmogorov–Sinai entropy production rate of the Markov matrix $\mathbf{P}^{[i]}$, measured in nats per word.444The entropy production rate $\eta(\mathbf{P}):=-\sum_{i,j}\pi_{i}p_{ij}\log p_{ij}$ [9, (4.27)] of a Markov matrix $\mathbf{P}$ represents the weighted average (assigning probability mass $\pi_{i}$ to the ith Markov state) of Boltzmann’s partition entropies $-\sum_{j}p_{ij}\log p_{ij}$ [10, §8.2]. We have $\eta(\mathbf{P})\leq\log N$ for an $N\times N$ Markov matrix $\mathbf{P}$ with strictly positive entries[10, Theorem 14.1].

Via bipartite matching (Fig. 6) of word vectors $\mathbf{v}_{i}$ across languages, our algorithm translates words from parallel texts at very high precision (Fig. 5c), being competitive with state-of-the-art algorithms for bilingual word mapping [11, 12].

Unlike the vector $\mathbf{b}_{i}$ (Fig. 3c) that captures only chapter-scale features of $\mathsf{W}_{i}$, the semantic fingerprint $\mathbf{v}_{i}$ in (9) characterizes the kinetic behavior of $\mathsf{W}_{i}$ on all the long-range time scales.

Given a topical pattern $\mathsf{W}_{i}^{\mathrm{A}}$ in language A, its semantic fingerprint $\mathbf{v}_{i}^{\mathrm{A}}$ (a descending list of recurrence eigenvalues, as in Fig. 5b) allows us to numerically locate a semantically close pattern in a parallel text written in another language B, in two steps:

(1) Divide the document into $K$ chapters, and define the semantic similarity function as $s(\mathsf{W}_{i}^{\mathrm{A}},\mathsf{W}_{j}^{\mathrm{B}}):=s_{R}({\mathbf{v}}_{i}^{\mathrm{A}},{\mathbf{v}}_{j}^{\mathrm{B}})$ if

[TABLE]

(which is a ballpark screening more robust than Fig. 3c, with $\|\mathbf{b}\|_{0}$ counting the number of non-zero components in $\mathbf{b}$) and $s_{R}({\mathbf{v}}_{i}^{\mathrm{A}},{\mathbf{v}}_{j}^{\mathrm{B}})\geq 0.7$; $s(\mathsf{W}_{i}^{\mathrm{A}},\mathsf{W}_{j}^{\mathrm{B}}):=0$ otherwise.

(2) Solve a bipartite matching problem (Fig. 6) that maximizes $\sum_{i,j}s(\mathsf{W}_{i}^{\mathrm{A}},\mathsf{W}_{j}^{\mathrm{B}})$, using the Hungarian Method [13] attributed to Jacobi–Kőnig–Egerváry–Kuhn [14].

3.2 Machine-assisted text comprehension on WikiQA data set

By automatically discovering related words through numerical brainstorming (Figs. 5a and b, insets), our semantic cliques $\mathscr{S}_{i}$ are useful in text comprehension and question answering. We can expand a set of question words $\mathscr{Q}=\{\mathsf{W}_{q_{1}},\dots,\mathsf{W}_{q_{K}}\}$ into $\mathscr{Q}\cup\mathscr{Q}^{\prime}=\bigcup_{k=1}^{K}\mathscr{S}_{q_{k}}$, by bringing together the semantic cliques $\mathscr{S}_{q_{k}}$ generated from a reference text by each and every question word $\mathsf{W}_{q_{k}}$.

As before, we construct a localized Markov matrix $\mathbf{P}=(p_{ij})_{1\leq i,j\leq N}$ on this subset of word patterns $\mathscr{Q}\cup\mathscr{Q}^{\prime}$. We further use the Brin–Page damping [15] to derive an ergodic Markov matrix $\mathbf{\widetilde{P}}=(\widetilde{p}_{ij})_{1\leq i,j\leq N}$, where $\widetilde{p}_{ij}=0.85p_{ij}+\frac{0.15}{N}$.

By analogy to the behavior of internet surfing [15, 16], we model the process of associative reasoning [17] as a navigation through the nodes $\mathscr{Q}\cup\mathscr{Q}^{\prime}$ according to $\mathbf{\widetilde{P}}$, which quantifies the click-through rate from one idea to another. The PageRank recursion [16] ensures a unique equilibrium state $\bm{\widetilde{\pi}}$ attached to $\mathbf{\widetilde{P}}$. If our question $Q$ and a candidate answer $A$ contain, respectively, words from $\mathsf{W}_{Q_{1}}$, $\dots$, $\mathsf{W}_{Q_{m}}\in\mathscr{Q}$ and $\mathsf{W}_{A_{1}}$, $\dots$, $\mathsf{W}_{A_{n}}\in\mathscr{Q}\cup\mathscr{Q}^{\prime}$ (counting multiplicities, but excluding function words and patterns with fewer than 3 occurrences in the reference document), then we assign the following entropy production score

[TABLE]

to this question-answer pair.555One may compare the score $\mathscr{F}[Q,A]$ to the Kolmogorov–Sinai entropy production rate [9, (4.27)] $\eta(\mathbf{P})=-\sum_{i=1}^{N}\sum_{j=1}^{N}\pi_{i}p_{ij}\log p_{ij}$ of a Markov matrix $\mathbf{P}=(p_{ij})_{1\leq i,j\leq N}$. The score $\mathscr{F}[Q,A]$ is modeled after Boltzmann’s partition entropies, weighted by words in the question, and sifted by topics in the answer. Such a weighting and sifting method is analogous to the definition of scattering cross-sections in particle physics.

A sample work flow is shown in Fig. 7a, to illustrate how our rudimentary question-answering machine handles a query. To answer a question, we use a single Wikipedia page (without infoboxes and other structural data) as the only reference document and training source. Like a typical human reader of Wikipedia, our numerical associative reasoning generates a weighted set of nodes $\mathscr{Q}\cup\mathscr{Q}^{\prime}$ (presented graphically as a thought bubble in Fig. 7a), without the help of external stimuli or knowledge feed. Here, the relative weights in the nodes of $\mathscr{Q}\cup\mathscr{Q}^{\prime}$ are computed from the equilibrium state $\bm{\widetilde{\pi}}$ of $\mathbf{\widetilde{P}}$, via the PageRank algorithm.

We then test our semantic model (LISF in Fig. 7b) on all the 1242 questions in the WikiQA data set, each of which is accompanied by at least one correct answer located in a designated Wikipedia page. Our algorithm’s performance is roughly on par with LCLR and CNN benchmarks [18], improving upon the baseline by significant margin. This is perhaps remarkable, considering the relatively scant data at our disposal. Unlike the LCLR approach, our numerical discovery of synonyms does not draw on the WordNet database [19] or pre-existent corpora of question-answer pairs. Unlike the CNN method, we do not need pre-trained word2vec embeddings [20] as semantic input.

Moreover, our algorithm (LISF∗ in Fig. 7b) performs slightly better on a subset of 990 questions that do not require quantitative cues (How big? How long? How many? How old? What became of? What happened to? What year? and so on). This indicates that, with a Markov chain description of two-body interactions between topics, our structural model fits associative reasoning better than rule-based reasoning [17], while imitating human behavior in the presence of limited data. To enhance the reasoning capabilities of our algorithm, it is perhaps appropriate to apply a Markov random field [21, §4.1.3] to graphs of word patterns, to capture many-body interactions among different topics.

4 Conclusion

In our current work, we define semantics through algebraic invariants that are concept-specific and language-independent. To construct such invariants, we develop a stochastic model that assigns a semantic fingerprint (list of recurrence eigenvalues) to each concept via its long-range contexts. Consistently using a single Markov framework, we are able to extract topics (Figs. 2e, 3b,b’), translate topics (Figs. 3c, 4c, 5b,c, 6) and understand topics (Figs. 5a,b, 7a,b), through statistical mining of short and medium-length texts. In view of these three successful applications, we are probably close to a complete set of semantic invariants, after demystifying the long-range behavior of human languages.

Notably, our algorithms apply to documents of moderate lengths, similar to the experience of human readers. This contrasts with data-hungry algorithms in machine learning [18, 22], which utilize high-dimensional numerical representations of words and phrases [20, 23, 11, 12] from large corpora. Our semantic mechanism exhibits universality on long-range linguistic scales. This adds to our quantitative understanding of diversity on shorter-range linguistic scales, such as phonology [24, 25, 26], morphology [27, 28, 29, 30] and syntax [31, 32, 3, 33, 30].

Thanks to the independence between semantics and syntax [3], our current model conveniently ignores the non-Markovian syntactic structures which are essential to fluent speech. In the near future, we hope to extend our framework further, to incorporate both Markovian and non-Markovian features across different ranges. The Mathematical Principles of Natural Languages, as we envision, must and will combine the statistical analysis of a Markov model with linguistic properties on shorter time scales that convey morphological [27, 28, 29, 30] and syntactical [31, 32, 3, 33, 30] information.

Appendix A Poissonian banality and non-topicality

Here, we first present a proof of our statistical criterion for Poissonian banality (which we identify with non-topicality of word patterns), as stated in (1).

Theorem 1** (Sums and products of exponentially distributed random variables)**

Let $X_{1},\dots,X_{N}$ be independent random variables, each obeying an exponential distribution with mean 1. The probability distribution of

[TABLE]

is asymptotic to a Gaussian law

[TABLE]

as $N\to\infty$.

Proof:

By definition, we have a moment-generating function

[TABLE]

To evaluate the last multiple integral, we use the spherical coordinates $\xi_{1}=r\cos\theta_{1}$, $\xi_{2}=r\sin\theta_{1}\cos\theta_{2}$, $\xi_{3}=r\sin\theta_{1}\sin\theta_{2}\cos\theta_{3}$, $\dots$, $\xi_{N}=r\prod_{j=1}^{N-1}\sin\theta_{j}$ (where $r>0$, and $0<\theta_{j}<\pi/2$ for all $j\in\{1,\dots,N-1\}$), with volume element

[TABLE]

The result reads

[TABLE]

where $\Gamma(s):=\int_{0}^{\infty}x^{s-1}e^{-x}\operatorname{d}x$ is Euler’s gamma function.

Consequently, the proof of (13) builds on a cumulant expansion of (16), that is, development of $\log\mathbb{E}e^{-tY_{N}}$ up to $O(t^{2})$ terms.∎

If we have $N$ samples of recurrence times from a Poisson process, then the statistic $Y_{N}$ satisfies the inequality in (1) with probability $\int_{-2}^{2}e^{-{x^{2}}/{2}}\frac{\operatorname{d}x}{\sqrt{2\pi}}\approx 0.95$, in view of the theorem above.

Appendix B Recurrence times on an ergodic Markov chain with detailed balance

B.1 Background in probability theory

Based on numerical evidence (Fig. 4), we postulate that at the discourse level (the longest time scale in Friederici’s [4] neurobiological hierarchy), the production of natural language texts can be caricatured by the stochastic transitions on a stationary and ergodic Markov chain $\mathscr{M}=(\mathscr{S},\mathbf{P})$.666To reduce notational burden, we will not use superscripted asterisks to mark Markov matrices, beyond this point.

Here, the state space $\mathscr{S}=\{\mathsf{W}_{1},\dots,\mathsf{W}_{N}\}$ runs over finitely many word patterns occurring in the text, which in turn is representable as a discrete-time stochastic process $(X(0)$, $X(1)$, $\dots$, $X(n)$, $\dots)$; the transition matrix $\mathbf{P}=(p_{ij})$ describes the conditional hopping probability on the web of word patterns: $p_{ij}=\mathbb{P}(X(n+1)=\mathsf{W}_{j}|X(n)=\mathsf{W}_{i})$, for $n\in\mathbb{Z}_{\geq 0}$. Depending on context, we also model a document by a localized Markov chain $\mathscr{M}^{\prime}=(\mathscr{S}^{\prime},\mathbf{P}^{\prime})$, where certain word patterns $\mathsf{W}_{i}$ are removed from the state space $\mathscr{S}$ to form a proper subset $\mathscr{S}^{\prime}\subsetneqq\mathscr{S}$.

In a more formal setting, our notation $\mathscr{M}=(\mathscr{S},\mathbf{P})$ for the Markov chain should be expanded into $\mathscr{M}=(\varOmega,\mathscr{F},\{\mathbb{P}^{\mathsf{W}}:\mathsf{W}\in\mathscr{S}\},\{X(t):t\in\mathbb{Z}_{\geq 0}\},\{\widehat{\theta}_{n}:n\in\mathbb{Z}_{\geq 0}\})$, whose components are explained below.

• •

The sample space $\varOmega$ consists of all stochastic trajectories $\omega=(X(0),X(1),\dots,X(t),X(t+1),\dots)=(X(t))_{t\in\mathbb{Z}_{\geq 0}}$ on the state space $\mathscr{S}$, namely, all possible texts that can be analyzed by our particular model.

• •

Each member in the field of events $\mathscr{F}=2^{\varOmega}$ (the totality of all subsets in the sample space $\varOmega$) is a set containing zero or more stochastic processes that can be regarded as caricatures of text productions.

• •

The family of probability measures $\{\mathbb{P}^{\mathsf{W}}:\mathsf{W}\in\mathscr{S}\}$ are related to the transition matrix $\mathbf{P}=(p_{ij})_{1\leq i,j\leq N}$ by the identity $p_{ij}^{(t)}=\mathbb{P}^{\mathsf{W}_{i}}(X(t)=\mathsf{W}_{j})=\mathbb{P}(X(t)=\mathsf{W}_{j}|X(0)=\mathsf{W}_{i}),\forall\mathsf{W}_{i},\mathsf{W}_{j}\in\mathscr{S},\forall t\in\mathbb{Z}_{>0}$.

• •

The shift operator $\widehat{\theta}_{n}:\varOmega\longrightarrow\varOmega$ acts on an arbitrary trajectory $(X(0),X(1),\dots,X(t),X(t+1),\dots)=\omega\in\varOmega$ in the following manner: $\widehat{\theta}_{n}\circ\omega=(X(n),X(n+1),\dots,X(t+n),X(t+n+1),\dots),\forall t,n\in\mathbb{Z}_{\geq 0}$.

For discussions of the stopping times (a class of random variables on Markov chains) as well as the Markov property, it is convenient to further introduce the notation $\mathscr{F}_{n}$ for $n\in\mathbb{Z}_{\geq 0}$. Here, $\mathscr{F}_{n}$ is the smallest $\sigma$-algebra containing all the events in the form of $(X(0)=\mathsf{W}_{i_{0}},X(1)=\mathsf{W}_{i_{1}},\dots,X(n)=\mathsf{W}_{i_{n}})$. The family of $\sigma$-algebras $\{\mathscr{F}_{n}:n\in\mathbb{Z}_{\geq 0}\}$ forms a filtration: $\mathscr{F}_{n}\subseteq\mathscr{F}_{m}$ if $n\leq m$. For any $n\in\mathbb{Z}_{\geq 0}$, $B\in\mathscr{F}$, $\mathsf{W}\in\mathscr{S}$, we have the following relation concerning conditional expectations:

[TABLE]

(for every time-step $n\in\mathbb{Z}_{\geq 0}$ and Markov state $\mathsf{W}\in\mathscr{S}$) which is merely a reformulation of the Markov condition $\mathbb{P}^{\mathsf{W}}(X(n+1)=\mathsf{W}_{i_{n+1}}|X(n)=\mathsf{W}_{i_{n}},X(n-1)=\mathsf{W}_{i_{n-1}},\dots,X(0)=\mathsf{W}_{i_{0}})=\mathbb{P}^{\mathsf{W}_{i_{n}}}(X(1)=\mathsf{W}_{i_{n+1}})=\mathbb{P}^{\mathsf{W}}(X(n+1)=\mathsf{W}_{i_{n+1}}|X(n)=\mathsf{W}_{i_{n}})$. In other words, for any $A\in\mathscr{F}_{n}$, $B\in\mathscr{F}$, $\mathsf{W}\in\mathscr{S}$, we have the following statement of the Markov property:

[TABLE]

In both (17) and (17*′*), one can replace the indicator function $\mathbf{1}_{B}$ for event $B\in\mathscr{F}$ by any random variable with finite expectation.

B.2 Hitting time and return time on a Markov chain

We need to first precisely define the probability distributions for the hitting time and return time (the latter also known as “recurrence time”) on our Markov chain $\mathscr{M}=(\mathscr{S},\mathbf{P})$.

For each state $\mathsf{W}_{i}\in\mathscr{S}$ and trajectory $\omega=(X(t))_{t\in\mathbb{Z}_{\geq 0}}\in\varOmega$, we define

[TABLE]

then $\tau_{i}:\varOmega\longrightarrow\mathbb{Z}_{>0}\cup\{+\infty\}$ is a stopping time. Suppose that our pattern of interest corresponds to a subset of states $\mathscr{W}\subseteq\mathscr{S}$ on the web of words. We define another stopping time $\tau_{\mathscr{W}}:\varOmega\longrightarrow\mathbb{Z}_{>0}\cup\{+\infty\}$ as

[TABLE]

Clearly, $\tau_{\mathscr{W}}(\omega)$ is equal to the first time when a forward stepwise search lands on the set of interest $\mathscr{W}$. Recalling the invariant measure $\bm{\pi}=(\pi_{1},\dots,\pi_{N})$, we define the cumulative distribution function (CDF) for the hitting time to the set of patterns $\mathscr{W}$ as

[TABLE]

Similarly, the CDF for the return time to the set of patterns $\mathscr{W}$ is defined as

[TABLE]

In the next theorem, we present an identity that connects hitting and return times of Markov states, which in turn, is a discrete analog of the Haydn–Lacroix–Vaient relation for continuous-time ergodic dynamical systems [8].

Theorem 2** (Relation between hitting time and return time distributions)**

For a stationary and ergodic Markov chain $\mathscr{M}=(\mathscr{S},\mathbf{P})$, and a subset of states $\mathscr{W}\subseteq\mathscr{S}$, we have the following identity regarding the probability distribution of hitting and return times to $\mathscr{W}$:

[TABLE]

for all $t\in\mathbb{Z}_{>0}$.

Proof:

Clearly, our task is equivalent to the verification of the following formula:

[TABLE]

for all $t\in\mathbb{Z}_{>0}$.

For $t=1$, the left-hand side of (22*′*) can be computed as follows:

[TABLE]

This is equal to the right-hand side of (22*′*), because $\mathbb{P}^{\mathsf{W}_{j}}(\tau_{\mathscr{W}}<1)=0$.

For $t\in\mathbb{Z}_{>0}$, we can compute

[TABLE]

from the Markov property (17*′*). Here, we have $\sum_{\mathsf{W}_{i}\in\mathscr{S}}\mathbb{E}^{\mathsf{W}_{i}}(\mathbf{1}_{(X(1)\in\mathscr{W})})\pi_{i}=\sum_{\mathsf{W}_{i}\in\mathscr{S}}\mathbb{P}^{\mathsf{W}_{i}}(\tau_{\mathscr{W}}=1)\pi_{i}=\sum_{\mathsf{W}_{j}\in\mathscr{W}}\pi_{j}$ according to (23), and

[TABLE]

Therefore, the recursion

[TABLE]

for all $t\in\mathbb{Z}_{>0}$ allows us to build (22*′*) inductively on the $t=1$ case (23). ∎

B.3 Positive definite return time distributions on a Markov chain with detailed balance

Thanks to Theorem 2, our analysis of a return time distribution can be built on the analysis of the corresponding hitting time, the latter of which is usually easier to compute (in theoretical analysis).

For a stationary and ergodic Markov chain $(\mathscr{S},\mathbf{P}=(p_{ij})_{1\leq i,j\leq N})$, and a pattern of interest $\mathscr{W}\subsetneqq\mathscr{S}$, one can use the Markov property to show that the hitting time distribution satisfies

[TABLE]

and

[TABLE]

for all $t\in\mathbb{Z}_{>1}$.

Consider the abelian semigroup $\Sigma=\mathbb{Z}_{\geq 0}$. A semicharacter [34, p. 92, Definition 2.1] $\rho:\mathbb{Z}_{\geq 0}\longrightarrow\mathbb{C}$ must assume the following form: $\rho(0)=1$; $\rho(s)=[\rho(1)]^{s}$ for $s\in\mathbb{Z}_{>0}$. Since the spectral radius of our recurrence matrix is strictly less than 1, we will only concern ourselves with $\rho_{0}(s):=\mathbf{1}_{\{0\}}(s)$ and $\rho_{\lambda}(s):=\lambda^{s},s\in\mathbb{Z}_{>0}$ for $\lambda\in(-1,0)\cup(0,1)$.

According to the harmonic analysis on semigroups, a bounded function $f:\mathbb{Z}_{\geq 0}\longrightarrow\mathbb{C}$ can be represented as a weighted mixture of semicharacters $f(s)=\int_{-1<\lambda<1}\rho_{\lambda}(s)\operatorname{d}\mu(\lambda)$ if and only if it is positive definite [34, p. 93, Theorem 2.5]: the inequality

[TABLE]

holds for arbitrary positive integers $m\in\mathbb{Z}_{>0}$, and arbitrarily chosen $m$-dimensional “vectors” $(t_{1},\dots,t_{m})\in\Sigma^{m}$, $(c_{1},\dots,c_{m})\in\mathbb{C}^{m}$.

In the following theorem, we check the positive definiteness criterion (29) for the hitting time distribution $1-H_{\mathscr{W}}(s+2),s\in\mathbb{Z}_{\geq 0}$ on a Markov chain satisfying the detailed balance condition. This would imply that the return time distribution $1-R_{\mathscr{W}}(s+2),s\in\mathbb{Z}_{\geq 0}$ is also a weighted mixture of semicharacters $\rho_{\lambda}(s)$ for $-1<\lambda<1$, according to the finite difference relation (22) in Theorem 2.

Theorem 3** (Positive definite return time distributions)**

Suppose that a non-void pattern $\mathscr{W}\subsetneqq\mathscr{S}$ on a stationary and ergodic Markov chain $\mathscr{M}=(\varOmega,\mathscr{F},\{\mathbb{P}^{\mathsf{W}}:\mathsf{W}\in\mathscr{S}\},\{X(t):t\in\mathbb{Z}_{\geq 0}\},\{\widehat{\theta}_{n}:n\in\mathbb{Z}_{\geq 0}\})$ satisfies $\pi_{i}p_{ij}=\pi_{j}p_{ji}$ for all $\mathsf{W}_{i},\mathsf{W}_{j}\in\mathscr{S}\smallsetminus\mathscr{W}$. Then, the cumulative distribution for the return time to $\mathscr{W}$ has the following integral representation

[TABLE]

for a certain probability measure $\mathscr{P}_{\mathscr{W}}(\lambda)$ supported on $\lambda\in(-1,1)$.

Proof:

By (28), we have

[TABLE]

for $r,s\in\mathbb{Z}_{>0}$. We then apply the identity $\pi_{i}p_{ij}=\pi_{j}p_{ji},\forall\mathsf{W}_{i},\mathsf{W}_{j}\in\mathscr{S}\smallsetminus\mathscr{W}$ to the product over $n\in\mathbb{Z}\cap[1,r]$, which brings us

[TABLE]

for $r,s\in\mathbb{Z}_{>0}$. In other words, we have verified

[TABLE]

for $r,s\in\mathbb{Z}_{>0}$. Using the Markov property, one can readily generalize the identity above to $r,s\in\mathbb{Z}_{\geq 0}$. Thus, we have

[TABLE]

for $t_{r},t_{s}\in\mathbb{Z}_{\geq 0}$. By the positive definiteness criterion in (29), we see that $1-H_{\mathscr{W}}(s+2),s\in\mathbb{Z}_{\geq 0}$ is a weighted mixture of semicharacters. In view of the recursion identity relating hitting and return time distributions (26), we have confirmed the statement in (30). ∎

With bin sizes far larger than 2 time units on the Markov chain, our $L_{ii}$ is nearly a continuous random variable and all the period-2 oscillatory decays $\rho_{\lambda}(s)$ for $-1<\lambda<0$ behave just like noise terms in the histogram. Then, Theorem 3 tells us that the probability distribution for $L_{ii}$ can be approximated by a weighted mixture777We note that there exist exceptions to stationarity and ergodicity in realistic documents. A novel may involve the birth and/or death of leading/supporting characters. An academic treatise may place particular emphasis on certain concepts in specific chapters. In such scenarios, the overall recurrence kinetics may still follow the generic law given in (2), as a consequence of both detailed balance (in locally applied Markov models) and a heterogeneous mixture of several stationary and ergodic Markov models that patch together as a whole. of exponential decays $e^{-kL_{ii}}$, in the continuum limit.

B.4 *A statistical criterion for numerical independence between different word patterns

If we define $\widetilde{L}_{ij}$ as the effective length of a text fragment free from word pattern $\mathsf{W}_{j}$ that is simultaneously flanked by $\mathsf{W}_{i}$ to the left and by $\mathsf{W}_{j}$ to the right, then $\sum_{\mathsf{W}_{i}\in\mathscr{S}}\pi_{i}\mathbb{P}(\widetilde{L}_{ij}>t)\sim\sum_{\mathsf{W}_{i}\in\mathscr{S}}\pi_{i}\mathbb{P}^{\mathsf{W}_{i}}(\tau_{j}>t)$ represents the hitting time distribution to the Markov state $\mathsf{W}_{j}$. According to Theorem 2, the hitting time distribution can be obtained by integrating the return time distribution $\mathbb{P}(\widetilde{L}_{jj}>t)\sim\mathbb{P}^{\mathsf{W}_{j}}(\tau_{j}>t)\sim\sum_{n}c_{n}e^{-k_{n}t}$. Here, the positive weights $c_{n}>0$ are normalized to one: $\sum_{n}c_{n}=1$. In other words, for fixed $\mathsf{W}_{j}$, we can predict some statistical properties of $\widetilde{L}_{ij}$ by analyzing the recurrence statistic $\widetilde{L}_{jj}$, as described in the theorem below.

Theorem 4** (Mean and variance for the logarithm of hitting time)**

Using the continuous time approximation, we have the following relations for a fixed $\mathsf{W}_{j}$:

[TABLE]

where $\ell_{*}$ equals the right-hand side of (35), and the expectation $\mathbb{E}$ denotes average over all the states $\mathsf{W}_{i}$, weighted by $\pi_{i}$.

Proof:

In the continuous time approximation, we can assume that the probability density function for $\widetilde{L}_{jj}$ is $\sum_{n}c_{n}k_{n}e^{-k_{n}t}$, (where $c_{n},k_{n}>0$) so that the probability density function for $\widetilde{L}_{ij}$ (weighted average over all the states $\mathsf{W}_{i}$) is

[TABLE]

In view of (37), we can compute the left-hand side of (35) as

[TABLE]

Meanwhile, we may evaluate the right-hand side of (35) as follows:

[TABLE]

Thus, (35) is confirmed.

In a similar vein, we can argue that the left-hand side of (36) equals

[TABLE]

while the right-hand side of (36) amounts to

[TABLE]

This completes the proof of (36). ∎

The detailed balance condition $\pi_{i}p_{ij}=\pi_{j}p_{ji}$ means that a Markov chain is reversible, and the reverse chain has the same transition matrix $\mathbf{P}$ [35, §4.7, p. 203].

By the reversibility of our Markov chain, we may extend the results from the theorem above to a dual situation, namely, the statistical properties for $L_{ij}$, the effective length of a text fragment free from word $\mathsf{W}_{i}$ that is simultaneously flanked by $\mathsf{W}_{i}$ to the left and by $\mathsf{W}_{j}$ to the right (as considered in Fig. 1).

Trading $\widetilde{L}_{ij}$ (resp. $\widetilde{L}_{jj}$) in (35)–(36) for $L_{ij}$ (resp. $L_{ii}$), we recover (8): for a given i, we have used (35) as an estimate for $\langle\log L_{ij}\rangle$, and have used $1/n_{ij}$ times (36) as an estimate for the variance of $\langle\log L_{ij}\rangle$, which is taken over $n_{ij}$ independent samples.

Acknowledgments

We thank N. Chomsky and S. Pinker for their inputs on several problems of linguistics. We thank X. Sun for discussions on neural networks. We thank X. Wan, R. Yan and D. Zhao for their suggestions on experimental design, during the early stages of this work. We thank two anonymous reviewers, whose thoughtful comments help us improve the presentation of this work.