Spectral Simplicity of Apparent Complexity, Part II: Exact Complexities and Complexity Spectra

TL;DR

This paper develops a spectral decomposition framework for analyzing complex stochastic processes, providing exact formulas for various information-theoretic measures and revealing mathematical similarities among them.

Contribution

It introduces a novel spectral analysis method for hidden Markov models, enabling exact computation of complexity measures and exploring their interrelations.

Findings

Derived closed-form expressions for entropy-related measures.

Revealed mathematical similarities between different complexity measures.

Introduced coronal spectrograms for frequency-dependent analysis.

Abstract

The meromorphic functional calculus developed in Part I overcomes the nondiagonalizability of linear operators that arises often in the temporal evolution of complex systems and is generic to the metadynamics of predicting their behavior. Using the resulting spectral decomposition, we derive closed-form expressions for correlation functions, finite-length Shannon entropy-rate approximates, asymptotic entropy rate, excess entropy, transient information, transient and asymptotic state uncertainty, and synchronization information of stochastic processes generated by finite-state hidden Markov models. This introduces analytical tractability to investigating information processing in discrete-event stochastic processes, symbolic dynamics, and chaotic dynamical systems. Comparisons reveal mathematical similarities between complexity measures originally thought to capture distinct informational and computational properties. We also introduce a new kind of spectral analysis via coronal spectrograms and the frequency-dependent spectra of past-future mutual information. We analyze a number of examples to illustrate the methods, emphasizing processes with multivariate dependencies beyond pairwise correlation. An appendix presents spectral decomposition calculations for one example in full detail.

Figures

Equations

$$\gamma (L)$$

$$h_{\mu }(L)$$

$$A^L$$

$$+\left[0\in {\Lambda }_A\right]\sum _{m=0}^{{\nu }_0-1}{\delta }_{L,m}{A}_0{A}^m,$$

$$\left(\genfrac{}{}{0ex}{}{L}{m}\right)$$

$$\gamma (L)=⟨\pi \overline{\mathcal{A}}|T^{|L|-1}|\mathcal{A}1⟩,$$

$$⟨\pi \overline{\mathcal{A}}|$$

$$|\mathcal{A}1⟩$$

$$\gamma (L)$$

$$+[0\in {\Lambda }_T]\sum _{m=0}^{{\nu }_0-1}⟨\pi \overline{\mathcal{A}}|T_0{T}^m|\mathcal{A}1⟩{\delta }_{|L|-1,m}$$

$$={\gamma }_{⇝}(L)+{\gamma }_{⊸}(L).$$

$${\gamma }_{⇝}(L)$$

$$=⟨\pi \overline{\mathcal{A}}|T^{\mathcal{D}}{T}^{|L|}|\mathcal{A}1⟩,$$

$${\gamma }_{⊸}(L)=⟨\pi \overline{\mathcal{A}}|T_0{T}^{|L|-1}|\mathcal{A}1⟩,$$

$$h_{\mu }(L)$$

$$+\left[0\in {\Lambda }_W\right]\sum _{m=0}^{{\nu }_0-1}{\delta }_{L-1,m}⟨{\delta }_{\pi }|W_0{W}^m|H(W^{\mathcal{A}})⟩$$

$$=h_{⇝}(L)+h_{⊸}(L).$$

$$h_{⇝}(L)$$

$$=⟨{\delta }_{\pi }|W^{\mathcal{D}}{W}^L|H(W^{\mathcal{A}})⟩,$$

$$h_{⊸}(L)=⟨{\delta }_{\pi }|W_0{W}^{L-1}|H(W^{\mathcal{A}})⟩.$$

$$\underset{L\to \infty }{\lim }$$

$$=\underset{L\to \infty }{\lim }\sum _{\genfrac{}{}{0ex}{}{\lambda \in {\Lambda }_T}{\lambda \ne 0}}\sum _{m=0}^{{\nu }_{\lambda }-1}⟨\pi \overline{\mathcal{A}}|T_{\lambda ,m}|\mathcal{A}1⟩\left(\genfrac{}{}{0ex}{}{L-1}{m}\right){\lambda }^{L-1-m}$$

$$=⟨\pi \overline{\mathcal{A}}|T_1|\mathcal{A}1⟩$$

$$=⟨\pi \overline{\mathcal{A}}|1⟩⟨\pi |\mathcal{A}1⟩$$

$$\displaystyle=\Bigl{|}\sum_{{x}\in\mathcal{A}}{x}\Pr({x})\Bigr{|}^{2}$$

$$\displaystyle=\bigl{|}\braket{x}\bigr{|}^{2}~{}.$$

$$h_{\mu }$$

$$=\underset{L\to \infty }{\lim }\sum _{\genfrac{}{}{0ex}{}{\lambda \in {\Lambda }_W}{|\lambda |=1}}{\lambda }^{L-1}⟨{\delta }_{\pi }|W_{\lambda }|H(W^{\mathcal{A}})⟩$$

$$=⟨{\delta }_{\pi }|W_1|H(W^{\mathcal{A}})⟩,$$

$$h_{\mu }$$

$$\displaystyle=\bigl{\langle}h_{\mu}^{(\text{component }k)}\bigr{\rangle}_{k}~{}.$$

$$h_{\mu }$$

$$=⟨\pi ⟩H(T^{\mathcal{A}}).$$

$$⟨\pi ⟩H(T^{\mathcal{A}})=-\sum _{\sigma \in \mathcal{S}}\mathrm{Pr}(\sigma )\sum _{\begin{array}{c}x\in \mathcal{A}\\ {\sigma }^{\prime }\in \mathcal{S}\end{array}}{T}_{\sigma ,{\sigma }^{\prime }}^{(x)}{\log }_2(T_{\sigma ,{\sigma }^{\prime }}^{(x)}).$$

$$h_{\mu }(L)$$

$$=⟨{\delta }_{\pi }|W_1|H(W^{\mathcal{A}})⟩+\sum _{\genfrac{}{}{0ex}{}{\lambda \in {\Lambda }_W}{|\lambda |<1}}{\lambda }^{L-1}⟨{\delta }_{\pi }|W_{\lambda }|H(W^{\mathcal{A}})⟩,$$

$$\mathbf{E}$$

Full text

hidden Markov model, entropy rate, excess entropy, predictable information, statistical complexity, projection operator, complex analysis, resolvent, Drazin inverse

††preprint: Santa Fe Institute Working Paper 2017-06-XXX††preprint: arxiv.org:1706.XXXXX [nlin.cd]

1. I Introduction

2. II Correlation and Myopic Uncertainty

3. II.1 Nonasymptotics

4. II.2 Asymptotic correlation

5. II.3 Asymptotic entropy rate

6. III Accumulated Transients for Diagonalizable Dynamics

7. IV Exact Complexities and Complexity Spectra

8. IV.1 Excess entropy

9. IV.2 Persistent excess

10. IV.3 Excess entropy spectrum

11. IV.4 Synchronization information

12. IV.5 Power spectra

13. IV.6 Almost diagonalizable dynamics

14. IV.7 Markov order versus symmetry collapse

15. V Spectral Analysis via Coronal Spectrograms

16. VI Examples

17. VI.1 Golden Mean Processes

18. VI.2 Even Process

19. VI.3 Golden–Parity Process Family

20. VII Predicting Superpairwise Structure

21. VIII Conclusion

22. A Example Analytical Calculations

23. A.1 Process and spectra features

24. A.2 Observed correlation

25. A.3 Predictability

26. A.4 Synchronizing to predict optimally

**The prequel laid out a new toolset that allows one to analyze in detail how complex systems store and process information. Here, we use the tools to calculate in closed form almost all complexity measures for processes generated by finite-state hidden Markov models. Helpfully, the tools also give a detailed view of how subprocess components contribute to a process’ informational architecture. As an application, we show that the widely-used methods based on Fourier analysis and power spectra fail to capture the structure of even very simple structured processes. We introduce the spectrum of past-future mutual information and show that it allows one to detect such structure. **

I Introduction

Tracking the evolution of a complex system, a time series of observations often appears quite complicated in the sense of temporal patterns, stochasticity, and behavior that require significant resources to predict. Such complexity arises from many sources. Apparent complexity, even in simple systems, can be induced by practical measurement and analysis issues, such as small sample size, inadequate collection of probes, noisy or systematically distorted measurements, coarse-graining, out-of-class modeling, nonconvergent inference algorithms, and so on. The effects can either increase or decrease apparent complexity, as they add or discard information, hiding the system of interest from an observer to one degree or another. Assuming perfect observation, complexity can also be inherent in nonlinear stochastic dynamical processes—deterministic chaos, superexponential transients, high state-space dimension, nonergodicity, nonstationarity, and the like. Even in ideal settings, the smallest sufficient set of a system’s maximally predictive features is generically uncountable, making approximations unavoidable, in principle [1]. With nothing else said, these facts obviate physical science’s most basic goal—prediction—and, without that, they preclude understanding how nature works. How can we make progress?

The prequel, Part I, argued that this is too pessimistic a view. It introduced constructive results that address hidden structure and the challenges associated with predicting complex systems. Part I showed that questions regarding correlation, predictability, and prediction each require their own analytical structures, as long as one can identify a system’s hidden linear dynamic. It distinguished two genres of quantitative question: (i) cascading, in which the influence of an initial preparation cascades through state-space as time evolves, affecting the final measurement, and (ii) accumulating, in which statistics are gathered during such cascades. Part I identified the linear algebraic structure underlying each kind.

Part I explained that the hidden linear dynamic in systems induces a nondiagonalizable metadynamics, even if the dynamics are diagonalizable in their underlying state-space. Assuming normal and diagonalizable dynamics, so familiar in mathematical physics, simply fails in this setting. Thus, nondiagonalizable dynamics present an analytical roadblock. Part I reviewed a calculus for functions of nondiagonalizable operators—the recently developed meromorphic functional calculus of Ref. [2]—that directly addresses nondiagonalizability, giving constructive calculational methods and algorithms.

Along the way, Part I reviewed relevant background in stochastic processes and their complexities and the hidden Markov models (HMMs) that generate them. It delineated several classes of HMMs—Markov chains, unifilar HMMs, and nonunifilar HMMs. It also reviewed their mixed-state presentations (MSPs)—HMM generators of a process that track distributions induced by observation. Related constructions included the HMM and $\epsilon$-machine synchronizing MSPs, generator mixed-functional presentations, and cryptic-operator presentations. MSPs are key to calculating complexity measures within an information-theoretic framing. Part I then showed how each complexity measure reduces to a linear algebra of an appropriate HMM adapted to the cascading- or accumulating-question genre. It summarized the meromorphic functional calculus and several of its mathematical implications in relation to projection operators. Part I also highlighted a spectral weighted directed-graph theory that can give useful shortcuts for determining a process’ spectral decomposition. Part II here uses Part I’s notation and assumes familiarity with its results.

With Part I’s toolset laid out, Part II now derives the promised closed-form complexities of a process. Section §II investigates the range of possible behaviors for correlation and myopic uncertainty via convergence to asymptotic correlation and asymptotic entropy rates. Section §III then considers measures related to accumulating quantities during the transient relaxation to synchronization. Section §IV introduces closed-form expressions for a wide range of complexity measures in terms of the spectral decomposition of a process’ dynamic. It also introduces complexity spectra and highlights common simplifications for special cases, such as almost diagonalizable dynamics. Section §V gives a new kind of signal analysis in terms of coronal spectrograms. A suite of examples in §VI and §VII ground the theoretical developments and are complemented with an in-depth pedagogical example worked out in App. §A. Finally, we conclude with a brief retrospective of Parts I and II and give an eye towards future applications.

II Correlation and Myopic Uncertainty

Using Part I’s methods, our first step is to solve for the correlation function:

[TABLE]

and the myopic uncertainty or finite-history Shannon entropy rate:

[TABLE]

A comparison is informative. We then determine the asymptotic correlation and myopic uncertainty from the resulting finite-$L$ expressions.

II.1 Nonasymptotics

A central result in Part I was the spectral decomposition of powers of a linear operator $A$, even if that operator is nondiagonalizable. Recall that for any $L\in\mathbb{C}$:

[TABLE]

where $\binom{L}{m}$ is the generalized binomial coefficient:

[TABLE]

$\binom{L}{0}=1$, and $[0\in\Lambda_{A}]$ is the Iverson bracket. The latter takes on value $1$ if zero is an eigenvalue of $A$ and [math] if not.

In light of this, the autocorrelation function $\gamma(L)$ is simply a superposition of weighted eigen-contributions. Part I showed that Eq. (1) has the operator expression:

[TABLE]

where $T$ is the transition dynamic, $\mathcal{A}$ is the output symbol alphabet, and we defined the row vector:

[TABLE]

and the column vector:

[TABLE]

Substituting Part I’s spectral decomposition of matrix powers, Eq. (3) above, directly leads to the spectral decomposition of $\gamma(L)$ for nonzero integer $L$:

[TABLE]

We denote the persistent first term of Eq. (5) as $\gamma_{\rightsquigarrow}$, and note that it can be expressed:

[TABLE]

where $T^{\mathcal{D}}$ is $T$’s Drazin inverse. We denote the ephemeral second term as $\gamma_{\multimap}$, which can be written as:

[TABLE]

where $T_{0}$ is the eigenprojector associated with the eigenvalue of zero; $T_{0}=\bm{0}$ if $0\notin\Lambda_{T}$.

From Eq. (5), it is now apparent that the index of $T$’s zero eigenvalue gives a finite-horizon contribution ($\gamma_{\multimap}$) to the autocorrelation function. Beyond index $\nu_{0}$ of $T$, the only $L$-dependence comes via a weighted sum of terms of the form $\binom{|L|-1}{m}\lambda^{|L|-1-m}$—polynomials in $L$ times decaying exponentials. The set \bigl{\{}\bra{\pi\overline{\mathcal{A}}}T_{\lambda,m}\ket{\mathcal{A}\mathbf{1}}\bigr{\}} simply weights the amplitudes of these contributions. In the familiar diagonalizable case, the behavior of autocorrelation is simply a sum of decaying exponentials $\lambda^{|L|}$.

Similarly, in light of Part I’s expression for the myopic entropy rate in terms of the MSP—starting in the initial unsynchronized mixed-state $\pi$ and evolving the state of uncertainty via the observation-induced MSP transition dynamic $W$:

[TABLE]

—and its spectral decomposition of $A^{L}$, we find the most general spectral decomposition of the myopic entropy rates $h_{\mu}(L)$ to be:

[TABLE]

We denote the persistent first term of Eq. (8) as $h_{\rightsquigarrow}$, and note that it can be expressed directly as:

[TABLE]

where $W^{\mathcal{D}}$ is the Drazin inverse of the mixed-state-to-state net transition dynamic $W$. We denote the ephemeral second term as $h_{\multimap}$, which can be written as:

[TABLE]

From Eq. (8), we see that the index of $W$’s zero eigenvalue gives a finite horizon contribution ($h_{\multimap}$) to the myopic entropy rate. Beyond index $\nu_{0}$ of $W$, the only $L$-dependence comes via a weighted sum of terms of the form $\binom{L-1}{m}\lambda^{L-1-m}$—polynomials in $L$ times decaying exponentials. The set \bigl{\{}\bra{\delta_{\pi}}W_{\lambda,m}\ket{\operatorname{H}(W^{\mathcal{A}})}\bigr{\}} weights the amplitudes of these contributions.

For stationary processes we anticipate that, for all $\zeta\in\{\lambda\in\Lambda_{W}:|\lambda|=1,\lambda\neq 1\}$, $\bra{\delta_{\pi}}W_{\zeta}=\mathbf{0}$ and thus $\bra{\delta_{\pi}}W_{\zeta}\ket{\operatorname{H}(W^{\mathcal{A}})}=0$. Hence, we can save ourselves from superfluous calculation by excluding the nonunity eigenvalues on the unit circle, when calculating the myopic entropy rate for stationary processes. In the diagonalizable case, again, its behavior is simply a sum of decaying exponentials $\lambda^{L}$.

In practice, $\gamma_{\multimap}$ often vanishes, whereas $h_{\multimap}$ is often nonzero. This practical difference between $\gamma_{\multimap}$ and $h_{\multimap}$ stems from the difference between typical graph structures of the respective dynamics. For a stationary process’ generic transition dynamic, zero eigenvalues (and so $\nu_{0}(T)$ of $T$) typically arise from hidden symmetries in the dynamic. In contrast, the MSP of a generic transition dynamic often has tree-like ephemeral structures that are primarily responsible for the zero eigenvalues (and $\nu_{0}(W)$). Nevertheless, despite their practical typical differences, the same mathematical structures appear and contribute to the most general behavior of each of these cascading quantities.

The breadth of qualitative behaviors shared by autocorrelation and myopic entropy rate is common to the solution of all questions that can be reformulated as a cascading hidden linear dynamic; the myopic state uncertainty $\mathcal{H}^{+}(L)$ is just one of many other examples. As we have already seen, however, different measures of a process reflect signatures of different linear operators.

Next, we explore similarities in the qualitative behavior of asymptotics and discuss the implications for correlation and entropy rate.

II.2 Asymptotic correlation

The spectral decomposition reveals that the autocorrelation converges to a constant value as $L\to\infty$, unless $T$ has eigenvalues on the unit circle besides unity itself. This holds if index $\nu_{0}$ is finite, which it is for all processes generated by finite-state HMMs and also many infinite-state HMMs. If unity is the sole eigenvalue with magnitude one, then all other eigenvalues have magnitude less than unity and their contributions vanish for large enough $L$. Explicitly, if $\operatorname*{argmax}_{\lambda\in\Lambda_{T}}|\lambda|=\{1\}$, then:

[TABLE]

This used the fact that $\nu_{1}=1$ and that $T_{1}=\ket{\mathbf{1}}\bra{\pi}$ for an ergodic process.

If other eigenvalues in $\Lambda_{T}$ besides unity lie on the unit circle, then the autocorrelation approaches a periodic sequence as $L$ gets large.

II.3 Asymptotic entropy rate

By the Perron–Frobenius theorem, $\nu_{\lambda}=1$ for all eigenvalues of $W$ on the unit circle. Hence, in the limit of $L\to\infty$, we obtain the asymptotic entropy rate for any stationary process:

[TABLE]

since, for stationary processes, $\bra{\delta_{\pi}}W_{\zeta}=\mathbf{0}$ for all $\zeta\in\{\lambda\in\Lambda_{W}:|\lambda|=1,\lambda\neq 1\}$. For nonstationary processes, the limit may not exist, but $h_{\mu}$ may still be found in a suitable sense as a function of time. If the process has only one stationary distribution over mixed states, then $W_{1}=\ket{\mathbf{1}}\bra{\pi_{W}}$ and we have:

[TABLE]

where $\pi_{W}$ is the stationary distribution over $W$’s states, found either from $\bra{\pi_{W}}=\bra{\delta_{\pi}}W_{1}$ or from solving $\bra{\pi_{W}}W=\bra{\pi_{W}}$.

A simple but interesting example of when ergodicity does not hold is the multi-armed bandit problem [3, 4]. In this, a realization is drawn from an ensemble of differently biased coins or, for that matter, over any other collection of IID processes. More generally, there can be many distinct memoryful stationary components from which a given realization is sampled, according to some probability distribution. With many attracting components we have the stationary mixed-state eigenprojector $W_{1}=\sum_{k=1}^{a_{1}}\ket{1_{k}}\bra{1_{k}}$, with $\braket{1_{j}}{1_{k}}=\delta_{j,k}$, where the algebraic multiplicity $a_{1}(T)=a_{1}(W)$ of the ‘1’ eigenvalue is the number of attracting components. The entropy rate becomes:

[TABLE]

Above, $\bra{\delta_{\pi}}1_{k}\rangle$ is the probability of ending up in component $k$, while $\langle 1_{k}\ket{\operatorname{H}(W^{\mathcal{A}})}$ is component $k$’s entropy rate. Thus, if nonergodic, the process’ entropy rate may not be the same as the entropy of any particular realization. Rather, the process’ entropy rate is a weighted average of those for the ensemble of sequences constituting the process.

For unifilar $\mathcal{M}$, the topology, transition probabilities, and stationary distribution over the recurrent states are the same for both $\mathcal{M}$ and its $\mathscr{S}$-MSP. Hence, for unifilar $\mathcal{M}$ we have:

[TABLE]

One can easily show that Eq. (16) is equivalent to the well-known closed-form expression for $h_{\mu}$ for unifilar presentations:

[TABLE]

For nonunifilar presentations, however, we must use the more general result of Eq. (13). This is similar to the calculation in Eq. (17), but must be performed over the recurrent states of a mixed-state presentation, which may be countable or uncountable.

III Accumulated Transients for Diagonalizable Dynamics

In the diagonalizable case, autocorrelation, myopic entropy rate, and myopic state uncertainty reduce to a sum of decaying exponentials. Correspondingly, we can find the power spectrum, excess entropy, and synchronization information respectively via geometric progressions.

For example, if $W$ is diagonalizable and has no zero eigenvalue, then the myopic entropy rate reduces to:

[TABLE]

where $\bra{\delta_{\pi}}W_{1}\ket{\operatorname{H}(W^{\mathcal{A}})}$ is identifiable as the entropy rate $h_{\mu}$.

It then follows that the excess entropy, which is the mutual information ${\bf E}=\operatorname{I}[\smash{\overleftarrow{{X}}};\smash{\overrightarrow{{X}}}]$ between the past and the future, is:

[TABLE]

Note that larger eigenvalues (closer to unity magnitude) drive the denominator $1-\lambda$ closer to zero and, thus, increase $\frac{1}{1-\lambda}$. Hence, larger eigenvalues—controlling modes of the mixed-state transition matrix that decay slowly—have the potential to contribute most to excess entropy. Small eigenvalues—quickly decaying modes—do not contribute. Putting aside the language of eigenvalues, one can paraphrase: slowly decaying transient behavior (of the distribution of distributions over process states) has the most potential to make a process appear complex.

Continuing, the transient information is:

[TABLE]

We now see that the transient information is very closely related to the excess entropy, differing only via the square in the denominators. This comparison between ${\bf E}$ and ${\bf T}$ closed-form expressions suggests an entire hierarchy of informational quantities based on eigenvalue weighting.

Performing a similar procedure for the synchronization information ${\bf S}^{\prime}$ shows that:

[TABLE]

The expressions reveal a remarkably close relationship between ${\bf S}^{\prime}$ and ${\bf E}$. Define $\bra{\cdot}\equiv\sum_{L=0}^{\infty}\bra{\delta_{\pi}}W^{L}$. Then:

[TABLE]

The relationship is now made plain:

[TABLE]

Although a bit more cumbersome, perhaps better intuition emerges if we rewrite $\bra{\cdot}$ as $\bra{\int\Pr(\eta,L)dL}$.

Again, large eigenvalues—slowly decaying modes of the mixed-state transition matrix—can make the largest contribution to synchronization information; small eigenvalues correspond to quickly decaying modes that do not have the opportunity to contribute. In fact, the potential of large eigenvalues to make large contributions is a recurring theme for many questions one has about a process. Simply stated, long-term behavior—what we often interpret as “complex” behavior—is dominated by a process’s largest-eigenvalue modes.

That said, a word of warning is in order. Although large-eigenvalue modes have the most potential to make contributions to a process’s complexity, the actual set of largest contributors also depends strongly on the amplitudes $\{\bra{\delta_{\pi}}W_{\lambda}\ket{\dots}\}$, where $\ket{\dots}$ is some quantifier vector of interest; e.g., $\ket{\dots}=\ket{\operatorname{H}[\eta]}$, $\ket{\dots}=\ket{\operatorname{H}(W^{\mathcal{A}})}$, or $\ket{\dots}=\ket{\mathbf{1}}$.

Hence, there is as-yet unanticipated similarity between ${\bf E}$ and ${\bf T}$ and another between ${\bf E}$ and ${\bf S}^{\prime}$—at least assuming diagonalizability. We would like to know the relationships between these quantities more generally. However, deriving the general closed-form expressions for accumulated transients is not tractable via the current approach. Rather, to derive the general results, we deploy the meromorphic functional calculus directly at an elevated level, as we now demonstrate.

IV Exact Complexities and Complexity Spectra

We now derive the most general closed-form solutions for several complexity measures, from which expressions for related measures follow straightforwardly. This includes an expression for the past–future mutual information or excess entropy, identifying two distinct persistent and transient components, and a novel extension of excess entropy to temporal frequency spectra components. We also give expressions for the synchronization information and power spectra. We explicitly address the class—a common one we argue—of almost diagonalizable dynamics. The section finishes by highlighting finite-order Markov order processes that, rather than being simpler than infinite Markov order processes, introduce technical complications that must be addressed.

Before carrying this out, we define several useful objects. Let $\rho(A)$ be the spectral radius of matrix $A$:

[TABLE]

For stochastic $W$, since $\rho(W)=1$, let $\Lambda_{\rho(W)}$ denote the set of eigenvalues with unity magnitude:

[TABLE]

We also define:

[TABLE]

and

[TABLE]

Eigenvalues with unity magnitude that are not themselves unity correspond to perfectly periodic cycles of the state-transition dynamic. By their very nature, such cycles are restricted to the recurrent states. Moreover, we expect the projection operators associated with these cycles to have no net overlap with the start-state of the MSP. So, we expect:

[TABLE]

for all $\lambda\in\Lambda_{\rho(W)}\setminus\{1\}$. Hence:

[TABLE]

We will also use the fact that, since $\rho(\mathcal{Q})<1$:

[TABLE]

and furthermore:

[TABLE]

as a consequence of Eq. (21) and our spectral decomposition.

Having seen complexity measures associated with prediction all take on a similar form in terms of the $\mathscr{S}$-MSP state-transition matrix, we expect to encounter similar forms for generically nondiagonalizable state-transition dynamics.

IV.1 Excess entropy

We are now ready to develop the excess entropy in full generality. Our tools turn this into a direct calculation. We find:

[TABLE]

Note that $(I-Q)^{-1}=\text{inv}(I-Q)$ here, since unity is not an eigenvalue of $Q$. Indeed, the unity eigenvalue was explicitly extracted from the former matrix to make an invertible expression.

For an ergodic process, where $W_{1}=\ket{\mathbf{1}}\bra{\pi_{W}}$, this becomes:

[TABLE]

Computationally, Eq. (23) is wonderfully useful. However, the subtraction of $h_{\mu}$ is at first mysterious. Especially so, when compared to the compact result for the excess-entropy spectral decomposition in the diagonalizable case given by Eq. (18).

Let’s explore this. Recall that Ref. [2] showed:

[TABLE]

for any stochastic matrix $T$. From this, we see that the general solution for ${\bf E}$ takes on its most elegant form in terms of the Drazin inverse of $I-W$:

[TABLE]

Recall too Part I’s explicit spectral decomposition:

[TABLE]

From this and Eq. (25), we see that the past–future mutual information—the amount of the future that is predictable from the past—has the general spectral decomposition:

[TABLE]

IV.2 Persistent excess

In light of Eq. (9), we see that there are two qualitatively distinct contributions to the excess entropy ${\bf E}={\bf E}_{\rightsquigarrow}+{\bf E}_{\multimap}$. One comprises the persistent leaky contributions from all $L$:

[TABLE]

and the other is a completely ephemeral piece that contributes only up to $W$’s zero-eigenvalue index $\nu_{0}$:

[TABLE]

IV.3 Excess entropy spectrum

Equation (25) immediately suggests that we generalize the excess entropy, a scalar complexity measure, to a complexity function with continuous part defined in terms of the resolvent—say, via introducing the complex variable $z$:

[TABLE]

Such a function not only monitors how much of the future is predictable, but also reveals the time scales of interdependence between the predictable features within the observations. Directly taking the $z$-transform of $h_{\mu}(L)$ comes to mind, but this requires tracking both real and imaginary parts or, alternatively, both magnitude and complex phase. To ameliorate this, we employ a transform of a closely related function that contains the same information.

Before doing so, we should briefly note that ambiguity surrounds the appropriate excess-entropy generalization. There are many alternate measures that approach the excess entropy as frequency goes to zero. For example, directly calculating from the meromorphic functional calculus, letting $z=e^{i\omega}$ we find:

[TABLE]

We are challenged, however, to interpret the fact that $\text{Re}\bra{\delta_{\pi}}(e^{i\omega}I-W)^{-1}\ket{\operatorname{H}(W^{\mathcal{A}})}+\frac{1}{2}h_{\mu}$ is not necessarily positive at all frequencies. Another direct calculation shows that:

[TABLE]

Enticingly, $\text{Re}\bra{\delta_{\pi}}e^{i\omega}(e^{i\omega}I-W)^{-1}\ket{\operatorname{H}(W^{\mathcal{A}})}-\frac{1}{2}h_{\mu}$ appears to be positive over all frequencies for all examples checked. It is not immediately clear which, if either, is the appropriate generalization, though. Fortunately, the Fourier transform of a two-sided myopic-entropy convergence function makes our upcoming definition of $\mathcal{E}(\omega)$ interpretable and of interest in its own right.

Let

h

h be the two-sided myopic entropy convergence function defined by:

[TABLE]

For stationary processes, it is easy to show that $\operatorname{H}({X}_{0}|{X}_{-L+1:0})=\operatorname{H}({X}_{0}|{X}_{1:L})$, with the result that

h

h is a symmetric function. Moreover,

h

h then simplifies to:

[TABLE]

where $h_{\mu}(0)\equiv\log_{2}(\mathcal{A})$ and, as before, $h_{\mu}(L)=\operatorname{H}({X}_{L}|{X}_{1:L})$ for $L\geq 1$ with $h_{\mu}(1)=\operatorname{H}({X}_{1})$.

The symmetry of the two-sided myopic entropy convergence function

h

h guarantees that its Fourier transform is also real and symmetric. Explicitly, the continuous part of the Fourier transform turns out to be:

[TABLE]

a strictly real and symmetric function of the angular frequency $\omega$. Here, $\bm{R}$ is the redundancy of the alphabet $\bm{R}\equiv\log_{2}|\mathcal{A}|-h_{\mu}$, as in Ref. [5].

The transform $\widetilde{\reflectbox{h}\text{h}}$ also has a discrete impulsive component. For stationary processes this consists solely of the Dirac delta function at zero frequency:

[TABLE]

Recall that the Fourier transform of a discrete-domain function is $2\pi$-periodic in the angular frequency $\omega$. This delta function is associated with the nonzero offset of the entropy convergence curve of positive-entropy-rate processes. The full transform is:

[TABLE]

Direct calculation using the meromorphic functional calculus of Ref. [2] shows that:

[TABLE]

This motivates introducing the excess-entropy spectrum $\mathcal{E}(\omega)$:

[TABLE]

The excess-entropy spectrum rather directly displays important frequencies of apparent entropy reduction. For example, leaky period-$5$ processes have a period-$5$ signature in the excess entropy spectrum.

As with its predecessors, the excess-entropy spectrum also has a natural decomposition into two qualitatively distinct components:

[TABLE]

The excess-entropy spectrum gives an intuitive and concise summary of the complexities associated with a process’ predictability. For example, given a graph of the excess entropy spectrum, the past–future mutual information can be read off as the height of the continuous part of the function as it approaches zero frequency:

[TABLE]

Indeed, the limit of zero frequency is necessary due to the delta function in the Fourier transform at exactly zero frequency:

[TABLE]

Reflecting on this, the delta function indicates one of the reasons the excess entropy has been difficult to compute in the past. This also sheds light on the role of the Drazin inverse: It removes the infinite asymptotic accumulation, revealing the transient structure of entropy convergence.

We also have a spectral decomposition of the excess-entropy spectrum:

[TABLE]

where, in the last equality, we assume that $W_{0}$ is real. This shows that, in addition to the contribution of typical leaky modes of decay in entropy convergence, the zero-eigenvalue modes contribute uniquely to the excess entropy spectrum. In addition to Lorentzian-like spectral curves contributed by leaky periodicities in the MSP, the excess-entropy spectrum also contains sums of cosines up to a frequency controlled by index $\nu_{0}$, which corresponds to the depth of the MSP’s nondiagonalizability. This is simply the duration of ephemeral synchronization in the time domain.

IV.4 Synchronization information

Once expressed in terms of the $\mathscr{S}$-MSP transition dynamic, the derivation of the excess synchronization information ${\bf S}^{\prime}$ closely parallels that of the excess entropy, only with a different ket $\ket{\cdot}$ appended. We calculate, as before, finding:

[TABLE]

For an ergodic process where $W_{1}=\ket{\mathbf{1}}\bra{\pi_{W}}$, this becomes:

[TABLE]

From Eq. (24), we see that the general solution for ${\bf S}^{\prime}$ takes on its most elegant form in terms of the Drazin inverse of $I-W$:

[TABLE]

From Eq. (32) and Eq. (26), we also see that the excess synchronization information has the general spectral decomposition:

[TABLE]

Again the form of Eq. (32) suggests generalizing synchronization information from a complexity measure to a complexity function $\mathscr{S}(\omega)$. In this case, the result is simply related to the Fourier transform of the two-sided myopic state-uncertainty $\mathcal{H}(L)$.

IV.5 Power spectra

The extended complexity functions, $\mathcal{E}(\omega)$ and $\mathscr{S}(\omega)$ just introduced, give the same intuitive understanding for entropy reduction and synchronization respectively as the power spectrum $P(\omega)$ gives for pairwise correlation. Recall that the power spectrum can be written as:

[TABLE]

We see that $\left(e^{i\omega}I-T\right)^{-1}$ is the resolvent of $T$ evaluated along the unit circle $z=e^{i\omega}$ for $\omega\in[0,2\pi)$. Hence, by Part I’s decomposition of the resolvent, the general spectral decomposition of the continuous part of the power spectrum is:

[TABLE]

As with $\mathcal{E}(\omega)$ and $\mathscr{S}(\omega)$, all continuous frequency dependence of the power spectrum again lies simply and entirely in the denominator of the above expression.

Analogous to Ref. [6]’s results, the power-spectrum delta functions arise from the eigenvalues of $T$ that lie on the unit circle:

[TABLE]

where $\omega_{\lambda}$ is related to $\lambda$ by $\lambda=e^{i\omega_{\lambda}}$. An extension of the Perron–Frobenius theorem guarantees that the eigenvalues of $T$ on the unit circle have index $\nu_{\lambda}=1$.

Together, these equations yield structural constraints via particular functional forms that are key to solving the inverse problem of inferring process models from measured data.

IV.6 Almost diagonalizable dynamics

The nondiagonalizability that appears most commonly in prediction metadynamics is of a special form that we call almost diagonalizable: when all eigenspaces except one—usually that associated with $\lambda=0$—are diagonalizable subspaces. In the current setting, we say that a matrix is almost diagonalizable if all of its eigenvalues with magnitude greater than zero have geometric multiplicity equal to their algebraic multiplicity.

Definition 1**.**

$W$* is almost diagonalizable if and only if $g_{\lambda}=a_{\lambda}$ for all $\lambda\in\Lambda_{W}^{\setminus 0}\equiv\,\Lambda_{W}\setminus\{0\}$.*

Fortunately, we treat such nondiagonalizability straightforwardly using $W^{L}$’s spectral decomposition for singular matrices. First off, Eq. (3) simplifies to:

[TABLE]

Then, to obtain the projection operators associated with each eigenvalue in $\Lambda_{W}^{\setminus 0}$ for an almost diagonalizable matrix $W$, we use Part I’s expression for operators with index-one eigenvalues with $\nu_{\lambda}=1$ for all $\lambda\in\Lambda_{W}^{\setminus 0}$. Finding:

[TABLE]

for each $\lambda\in\Lambda_{W}^{\setminus 0}$. Or, when more convenient in a calculation, we let $\nu_{0}\to a_{0}-g_{0}+1$ or even $\nu_{0}\to a_{0}$ in Eq. (35), since multiplying $W_{\lambda}$ by $W/\lambda$ has no effect.

With the set of projection operators $W_{\lambda}$ for all $\lambda\in\Lambda_{W}^{\setminus 0}$ in hand, we can use the fact from Part I that projection operators sum to the identity to determine the projection operator associated with the zero eigenvalue:

[TABLE]

This is sometimes simpler and easier to automate than evaluating $W_{0}$ via the methods of symbolic inversion and residues or via finding all left and right eigenvectors and generalized eigenvectors.

Almost diagonalizable metadynamics play a prominent role in prediction for both processes of finite Markov order and for the much more general class of processes with broken partial symmetries that can be detected within a finite observation window—the processes of finite symmetry-collapse discussed next.

IV.7 Markov order versus symmetry collapse

What if zero is the only eigenvalue in the transient structure of a process’ MSP? That is, what if there are no loops in the $\mathscr{S}$-MSP transient structure? The associated processes turn out to have finite Markov order.

For processes with finite Markov order $R$—such as, those whose support is a subshift of finite type [7]—the entropy-rate approximates not only converge but also become equal to the true entropy rate when conditioning on long enough histories. Explicitly, for $\ell\geq R+1$ [5]:

[TABLE]

or, equivalently, for $L\geq R$:

[TABLE]

For a finite-order Markov process, all MSP transient states must have identically zero probability after $R$ time-steps. The only way to achieve this is if the $\mathscr{S}$-MSP’s transient structure is an acyclic directed graph with all probability density flowing away from the unique start-state down to the recurrent component. This means that all eigenvalues associated with the transient states are zero. Moreover, the index of the zero-eigenvalue of the $\epsilon$-machine’s $\mathscr{S}$-MSP is equal to the Markov order for finite Markov-order processes. That is, if $\Lambda_{W}\setminus\Lambda_{T}=\{0\}$, then:

[TABLE]

In contrast, for stochastic processes whose support is a strictly sofic subshift [7], the Markov order diverges, but $\nu_{0}$ can vanish or be finite or infinite. Yet, in either the finite-type or sofic case, $\nu_{0}$ still tracks the duration of exact state-space collapse within the transient dynamics of synchronization. This suggests that $\nu_{0}$ captures the index of broken symmetries for strictly sofic processes, in analogy to the Markov order for subshifts of finite type. The name symmetry-collapse captures the essence of $\nu_{0}$’s role in both cases.

Let’s explain. In the first $\nu_{0}$ time-steps, symmetries are broken that synchronize an observer to the process. For the simple period-two process $\dots 010101010\dots$ the “symmetry” that is broken is the degeneracy of possible phases—the [math] phase or the $1$ phase of the period-$2$ oscillation. Initially, without making a measurement the two phases are indistinguishable. After a single observation, though, the observer learns the phase and is completely synchronized to the process. Hence, $\nu_{0}=1$ for this order-1 Markov process. Simple periodic processes with larger periods have a longer time before the phase information is fully known; hence, their larger Markov order.

For the more complex strictly sofic processes, there may also be symmetries, such as phase information, that are completely broken within a finite amount of time. However, this is only part of the overall transient metadynamics of synchronization. And so, the symmetries completely broken within the symmetry-collapse epoch occur in addition to lingering state uncertainties about a strictly sofic process. As a practical matter, a process’ predictability is often substantially enhanced through the finite epoch of symmetry-collapse. This becomes apparent in the examples to follow.

V Spectral Analysis via Coronal Spectrograms

Coronal spectrograms are a broadly useful tool in visualizing complexity spectra, from power spectra to excess entropy spectra. They were recently introduced by Ref. [6] to demonstrate how diffraction patterns of chaotic crystals emanate from the eigenvalue spectrum of the hidden spatial dynamic of stacked modular layers.

Coronal spectrograms display any frequency-dependent measure $f(\omega)$ of a process wrapped around the unit circle while showing the eigenvalues $\Lambda_{T}$ of the relevant linear dynamic $T$ within the unit circle in the complex plane. Figure 1a gives an example. This is appropriate for discrete-domain (e.g., discrete-time or discrete-space) dynamics. For continuous-time dynamics, the coronal spectrogram unwraps into what we call the coronated horizon, via the familiar discrete-to-continuous conformal mapping of the inside of the unit circle of the complex plane to the left half of the complex plane [8]. Figure 1b displays a discrete-time version of the coronated horizon. Ultimately, either the coronal spectrogram or coronated horizon yield the same information and lend the same important lesson: the eigenvalues of the hidden linear dynamic control allowed system behaviors.

Coronal spectrograms demonstrate that complex systems behave according to the spectrum of their hidden linear dynamic. The relevant frequency-dependent measure $f(\omega)$ emanates from the nonzero eigenvalues of the hidden linear dynamic: the closer eigenvalues approach the unit circle, the sharper the observed peaks. At one extreme, one observes Bragg-like reflections (delta-function contributions) when the eigenvalues fall on the unit circle. The collection of diffuse peaks observed is a sum of Lorentzian-like and, what we might call, super-Lorentzian-like line profiles. Indeed, the Lorentzian-like line profiles are the discrete-time version of a Lorentzian curve. While the continuous-domain Lorentzian is given by \text{Re}\bigl{(}\frac{c}{\omega-\lambda}\bigr{)}, the continuous-to-discrete conformal mapping $\omega\to e^{i\omega}$ directly yields our discrete-domain analog \text{Re}\bigl{(}\frac{c}{e^{i\omega}-\lambda}\bigr{)}. The super-Lorentzian-like line profiles have the form \text{Re}\bigl{[}(\frac{c}{e^{i\omega}-\lambda})^{n}\bigr{]}.

Zero eigenvalues also contribute to $f(\omega)$, but only sinusoidal contributions of discrete increments from $\cos(\omega)$ up to $\cos(\nu_{0}\,\omega)$. Since these are qualitatively distinct from the super-Lorentzian contributions and do not emanate radially from the eigenvalues the same way contributions from nonzero eigenvalues do, coronal spectrograms are most useful for understanding the contributions of nonzero eigenvalues. Nevertheless, the two contributions can be usefully disentangled, as shown later.

We use both coronal spectrograms and coronated horizons to visualize various features in the examples to follow.

VI Examples

VI.1 Golden Mean Processes

To explore finite Markov order in relation to various complexity measures let’s consider the $(R\text{-}k)$-Golden Mean (GM) Processes 111Historically, the name “Golden Mean” derives from the $(1\text{-}1)$-Golden Mean Process that forbids two consecutive {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1}s. It describes the symbolic dynamics of the chaotic one-dimensional shift map over the unit interval with generating partition, when the slope of the map is the golden mean $\phi=1+1/\phi=(1+\sqrt{5})/2$ [11]. Moreover, consistency with the stationary stochastic process generated implies $1-p=1/\phi$. Generalizing to arbitrary $p$ and especially to arbitrary $R$ and $k$ takes us away from the eponymous property into a setting sufficiently general to study the generic behavior of order-$R$ Markov processes and the complexities of predicting them.. This process family describes a unique transition-parametrized process for each Markov-order R\in\bigl{\{}\nu\in\mathbb{Z}:\nu\geq 1\bigr{\}} and each cryptic-order k\in\bigl{\{}\kappa\in\mathbb{Z}:1\leq\kappa\leq R\bigr{\}}. The $\epsilon$-machine for the $(4\text{-}3)$-Golden Mean Process is shown in Fig. 2a. From this the construction of all other $(R\text{-}k)$-Golden Mean processes can be discerned. In words, $(R\text{-}k)$-Golden Mean Processes are binary with alphabet \mathcal{A}=\{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0},{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1}\} and if the most recent history consists of at least $k$ consecutive {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0}s (and no {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1}s since then) then there is a probability $p$ of next observing a {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1} and a probability $1-p$ of simply seeing another {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0}. This entails $R$ consecutive {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1}s followed by at least $k$ consecutive {\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0}s.

The eigenvalues of the internal state-to-state transition matrix of the $\epsilon$-machine’s recurrent component are:

[TABLE]

In the limit of $p\to 1$, all $(R\text{-}k)$-Golden Mean Processes become perfectly periodic. In this limit, the eigenvalues are evenly distributed on the unit circle:

[TABLE]

At the other extreme, as $p\to 0$, all eigenvalues evolve to zero, except the stationary eigenvalue at $z=1$. At any setting of $p$, the nonunity eigenvalues lie approximately on a circle within the complex plane whose radius decreases nonlinearly from $1$ to [math] as $p$ is swept from $1$ to [math]. Simultaneously, this circle’s center moves from the origin to a positive real value and back to the origin as $p$ is swept from $1$ to [math]. Figure 3 shows how the eigenvalues of the $(5\text{-}3)$-Golden Mean Process evolve over the full range of $p$ as it sweeps from $1$ to [math].

In contrast to the $p$-dependent spectrum of the recurrent structure just discussed, the only eigenvalue corresponding to the transient structure of the $\mathscr{S}$-MSP is equal to zero, regardless of the transition parameter $p$. Recall that this is necessarily true for any process with finite Markov order. Hence, $\Lambda_{\mathcal{W}}=\Lambda_{T}\cup\{0\}$, with $\nu_{0}(\mathcal{W})=R$. The cryptic structure is similar: $\Lambda_{\zeta}=\Lambda_{T}\cup\{0\}$, with $\nu_{0}(\zeta)=k$, where $\zeta$ is the state-to-state transition matrix of the cryptic operator presentation.

Table 1 compares the $\epsilon$-machines, autocorrelation, power spectra, MSPs, myopic entropy rates, and myopic state uncertainties for three $p$-parametrized examples of $(R\text{-}k)$-GM processes.

The autocorrelation of each process captures their ‘leaky periodic’ behaviors: The leakiness originates from the self-transition at state {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}A} that adds a phase-slip noise to otherwise $(R+k)$–periodic behavior. Moreover, each process’ phase, and so its $\epsilon$-machine’s internal state, is uniquely identified after $R$ observations. This corresponds to the depth of the $\mathscr{S}$-MSP tree-like structure $\nu_{0}(\mathcal{W})=R$, the convergence of the myopic entropy rate $h_{\mu}(L)$ to the true entropy rate $h_{\mu}$ when conditioning on observations of finite block-length $L-1=R$, and the complete loss of causal state uncertainty $\mathcal{H}(L)$ after $L=R$ observations.

A paradigm of finite Markov order, the $(5\text{-}3)$-Golden Mean Process has a strictly tree-like structure in its MSP’s transients, which have a maximum depth equal to both $\nu(\mathcal{W})$ and its Markov order of $5$.

These analyses illustrate the typical behaviors of complexity measures for finite Markov-order processes. We next investigate examples of infinite Markov-order processes to draw attention to the characteristic differences of nonzero eigenvalues in their MSP transient structures.

VI.2 Even Process

The Even Process, shown in the first column of Table 2, is a well known example of a stochastic process that cannot be generated by any finite Markov-order approximation, yet it is described by a simple two-state HMM.

Infinite Markov order, in this case, stems from the fact that the process generates only an even number of consecutive $1$s, between [math]s. The countably infinite set of Markov chain states necessary to track this parity reflects the infinite order. Moreover, the surplus entropy rate $h_{\mu}(L)-h_{\mu}$ incurred when using a finite order-$(L-1)$ Markov approximation vanishes only asymptotically, being the sum of decaying exponentials. (See Table 2.) This is in stark contrast to the myopic entropy rate for the finite Markov order processes of Table 1. For them $h_{\mu}(L)$ drops to $h_{\mu}$ exactly at $L=R+1$. Similarly, the average state uncertainty $\mathcal{H}(L)$ for infinite Markov processes converges only asymptotically—and with the same set of decay rates as $h_{\mu}(L)$—to its asymptotic value of [math]. (This curve is not shown in Table 2 for lack of space.) Such long-lived decay is driven by nonzero eigenvalues in the $\mathscr{S}$-MSP transient structure.

The Even Process is a relatively simple example of an infinite Markov-order process. As expected for infinite Markov-order, its MSP’s transient structure had nonzero eigenvalues. Generally, though, two ranges of contribution are to be expected in synchronization dynamics. The first is a finite-horizon contribution to the past-future mutual information, corresponding to completely ephemeral zero eigenvalues in the MSP’s transient structure. The second is an infinite-horizon contribution to the past–future mutual information, arising from nonzero eigencontributions.

VI.3 Golden–Parity Process Family

To further explore the nature of infinite Markov order processes, we introduce the $(\nu_{0}\text{-}k)$-Golden-Parity-$(P)$ Processes. This family subsumes and extends the examples analyzed so far. The role of each parameter is explained in Fig. 2b, which displays a state-transition diagram of the $(4\text{-}3)$-GP-$(3)$ Process’ $\epsilon$-machine.

If $P=1$, the family reduces to the $(\nu_{0}\text{-}k)$-Golden Mean Process family, with tunable Markov $R=\nu_{0}(W)$ and cryptic $k$ orders. That is, $(\nu_{0}(\mathcal{W})\text{-}k)$-GP-$(1)$ = $(\nu_{0}(\mathcal{W})\text{-}k)$-GM. However, the Markov order becomes infinite whenever $P>1$. In this case the index $\nu_{0}(\mathcal{W})$ of the $\mathscr{S}$-MSP’s zero-eigenvalue—which controls the finite duration necessary to resolve all broken symmetries—and the cryptic order $k$ can still be tuned independently. The Even Process considered earlier is the $(0\text{-}0)$-Golden-Parity-$(2)$ Process.

Three examples of $(\nu_{0}(\mathcal{W})\text{-}k)$-Golden-Parity-$(P)$ processes are analyzed in Table 2. The $\mathscr{S}$-MSP transient structure for the second two clarifies the difference between (i) the symmetry collapse associated with completely ephemeral transient states that are fully depleted of probability density after $\nu_{0}(\mathcal{W})$ time-steps and (ii) the long-lived leaky transients whose probability density only vanishes as more-refined ambiguity is resolved.

Examining the myopic entropy convergence $h_{\mu}(L)$, the effect of these distinct routes to synchronization on the predictability can be seen: The process is much more predictable, on average, after $\nu_{0}(\mathcal{W})$ time-steps. However, the average predictability of an infinite-Markov-order process continues to increase with increasing observation window, albeit with exponentially diminishing returns. In general, we showed that this asymptotic convergence occurs as a sum of decaying exponentials from diagonalizable subspaces and as the product of polynomials and exponentials in the case of nondiagonalizable structures associated with nonzero eigenvalues. The apparent oscillations under the exponential decays are completely described by the leaky periodicities of the eigenvalues in the transient belief states.

Finally, note that the excess entropy spectrum $\mathcal{E}(\omega)$ shows the frequency domain view of observation-induced predictability. ${\bf E}=\lim_{\omega\to 0}\mathcal{E}(\omega)$ is the total past–future mutual information, which is also the excess entropy observed before full synchronization. The $\nu_{0}(\mathcal{W})$ symmetry collapse contributes significantly and early to the total excess entropy of the last two examples. Whereas, the asymptotic tails of synchronization associated with leaky periodicity of particular transient states of uncertainty accumulate their contribution to excess entropy rather slowly.

In addition to new intuitions about convergence behaviors in stochastic processes, the general and broadly applicable theoretical results here allow novel numerical investigations and unprecedentedly-accurate analyses of infinite-Markov-order processes. As an example of the latter, let us summarize several of the exact results derived in App. A for the $(p,q)$-parametrized $(2\text{-}1)$-GP-$(2)$ process explored in Table 2’s second column.

Depending on whether the transition parameter $p$ is larger or smaller than $2\sqrt{q}-q$, App. A found qualitatively distinct behaviors dominate the $(2\text{-}1)$-GP-$(2)$ process. This hints at a general principle: behaviorally distinct regions are separated by a critical line in the $(p,q)$-parameter space along which the transition dynamic $T$ becomes nondiagonalizable. For $p<2\sqrt{q}-q$, the autocorrelation for $|L|\geq 2$ has the exact solution:

[TABLE]

where $\beta\equiv 2(p+2q)/(1+p+2q)$, $\zeta\equiv(\xi+1)^{2}(p\xi+2q)/(\xi(\xi^{3}+p\xi+2q))$, $\xi\equiv-\frac{1}{2}(p+q)+i\frac{1}{2}\sqrt{4q-(p+q)^{2}}$, and \omega_{\xi}\equiv\tfrac{\pi}{2}+\arctan\bigl{(}(p+q)/\sqrt{4q-(p+q)^{2}}\bigr{)}. The corresponding power spectrum is:

[TABLE]

For any parameter setting, the metadynamic of observation-induced synchronization to the $(2\text{-}1)$-GP-$(2)$ process is nondiagonalizable due to the index-$2$ zero eigenvalue. This leads to a completely ephemeral contribution to $h_{\mu}(L)$ up to $L=2$. For $L\geq 3$, we find the myopic entropy rate relaxes asymptotically to the true entropy rate according to:

[TABLE]

where the process’ true entropy rate is:

[TABLE]

Interestingly, while the autocorrelation at separation $L$ scales as $\sim q^{L/2}$, the predictability of single-symbol transitions between slightly shifted histories of length $L$ converges as $\sim p^{L/2}$—indicating two rather independent decay rates.

The amount of the future that can be predicted from the past is the total mutual information between the observable past and observable future:

[TABLE]

However, to actually perform prediction requires more memory than this amount of shared information. Calculation of additional measures and more detail can be found in App. A.

To explore the structure in infinite-cryptic order processes, one can use the more generalized family of $(\nu_{0}(\mathcal{W})\text{-}\nu_{0}(\zeta))$-GP-$(P\text{-}Z)$ Processes. For them, $\nu_{0}(\zeta)$ is the index of the zero-eigenvalue of the cryptic operator presentation and the process has infinite cryptic order whenever $Z>1$. Above, $Z=1$ and $(\nu_{0}(\mathcal{W})\text{-}\nu_{0}(\zeta))$-GP-$(P\text{-}1)$ = $(\nu_{0}(\mathcal{W})\text{-}\nu_{0}(\zeta))$-GP-$(P)$. Since the preceding examples served well enough to illustrate the power of spectral decomposition, our main goal, we leave a full analysis of this family to interested others.

VII Predicting Superpairwise Structure

The Random–Random–XOR (RRXOR) Process is generated by a simple HMM. Figure 4 displays its five-state $\epsilon$-machine. However, it illustrates nontrivial, counterintuitive features typical of stochastic dynamic information processing systems. The process is defined over three steps that repeat: (i) a \color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0 or \color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1 is output with equal probability, (ii) another \color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0 or \color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1 is output with equal probability, and then (iii) the eXclusive-OR operation (XOR) of the last two outputs is output.

Surprisingly, but calculations easily verify, there are no pairwise correlations. All of its correlations are higher than second order. One consequence is that its power spectrum is completely flat—the signature of white noise; see Fig. 5. This would lead a casual observer to incorrectly conclude that the generated time series has no structure. In fact, a white noise spectrum is an indication that, if structure it present, it must be hidden in higher-order correlations.

The RRXOR Process clearly is not structureless—via the exclusive OR, it transforms information in a substantial way. We show that the complexity measures introduced above can detect this higher-order structure. However, let us first briefly consider why the correlation-based measures fail to detect structure in the RRXOR Process.

It is sometimes noted that information measures are superior to standard measures of correlation since they capture nonlinear dependencies, while the standard correlation relies on linear models. And so, we can avoid this problem by using the information correlation $\operatorname{I}[X_{0};X_{\tau}]$ rather than autocorrelation. Analogous to autocorrelation, it too has a spectral version—the power-of-pairwise information (POPI) spectrum:

[TABLE]

It is easy to show that $\mathcal{I}(\omega)=0$ for the RRXOR Process. Hence, as Fig. 5 showed, such measures are not sufficient to detect even simple computational structure, since they only can detect pairwise statistical dependencies.

In stark contrast, the excess entropy spectrum $\mathcal{E}(\omega)$ does identify the structure of hidden dependencies in the RRXOR Process; see Fig. 6. Why? The brief detour through power spectra, information correlation, and POPI spectra brings us to a deeper understanding of why $\mathcal{E}(\omega)$ is successful at detecting nuanced computational structure in a time series. Since it partitions all random variables throughout time, the excess entropy itself picks up any systematic influence the past has on the future. The excess entropy spectrum further identifies the frequency decomposition of any such linear or nonlinear dependencies. In short, all multivariate dependencies contribute to the excess entropy spectrum.

Let us now consider the hidden structure of the RRXOR Process in more detail. With reference to Fig. 4, we observe that the expected probability density over causal states evolves through the $\epsilon$-machine with a period-$3$ modulation. In a given realization, the particular symbols emitted after each phase resetting ($\phi=0$) break symmetries with respect to which “wings” of the $\epsilon$-machine structure are traversed. This is reflected in $T$’s eigenvalues: the three roots of unity $\{e^{in2\pi/3}\}_{n=0}^{2}$ and two zero eigenvalues, with $a_{0}(T)=g_{0}(T)=2$ giving index $\nu_{0}(T)=1$.

The period-$3$ modulation leads to a phase ambiguity when an observer synchronizes to the process, an ambiguity that resolved in the MSP transient structure. This resolution is rather complicated, as made explicit in the RRXOR Process’ $\mathscr{S}$-MSP, shown in Fig. 7. There are $31$ transient states of uncertainty, in addition to the five recurrent states—$36$ causal states in total.

Since we derived the $\epsilon$-machine’s $\mathscr{S}$-MSP, $W=\mathcal{W}$. Hence, the MSP’s layout depicts the information processing involved while an observer synchronizes to the RRXOR Process. This graphically demonstrates the burden on an optimal predictor, even one that only needs to learn an average of $h_{\mu}$ bits per observation to optimally predict the process.

The MSP introduces new, relevant zero eigenvalues associated with its transient states. In particular, the first-encountered tree-like transients (starting with mixed-state $\pi$) introduce new Jordan blocks up to dimension $2$. Overall, the [math]-eigenspace of $W$ has index $2$, so that $\nu_{0}(W)=2$.

Two different sets of leaky-period-$3$ structures appear in the MSP transients. There are four leaky three-state cycles, each with the same leaky-period-3 contributions to the spectrum: \bigl{\{}(\tfrac{1}{4})^{1/3}e^{in2\pi/3}\bigr{\}}_{n=0}^{2}. There are also four leaky four-state cycles, each with a leaky-period-3 contribution and symmetry-breaking 0-eigenvalue contribution to the spectrum: \bigl{\{}(\tfrac{1}{2})^{1/3}e^{in2\pi/3}\bigr{\}}_{n=0}^{2}\cup\{0\}. The difference in eigenvalue magnitude, $(\tfrac{1}{4})^{1/3}$ versus $(\tfrac{1}{2})^{1/3}$, implies different timescales of synchronization associated with distinct learning tasks. For example, an immediate lesson is that it takes longer (on average) to escape the $4$-state leaky-period-$3$ components (from the time of arrival) than to escape the preceding $3$-state leaky-period-$3$ components of the synchronizing metadynamic.

The entropy rate convergence plots of Figs. 8 and 9 reveal a sophisticated predictability modulation that simply could not have been gleaned from the spectra of Fig. 5. Figure 9 emphasizes the dominance of the slowest-decaying eigenmodes for large $L$. Such oscillations under the exponential convergence to synchronization are typical. However, as seen in comparison with Fig. 8 much of the uncertainty may be reduced before this asymptotic mode comes to dominate. Ultimately, synchronization to optimal prediction may involve important contributions from all modes of the mixed-state-to-state metadynamic.

This detailed analysis of the RRXOR Process suggests several general lessons about how we view information in stochastic processes. First, as information processing increases in sophistication, a vanishing amount of a process’ intrinsic structure will be discernible at low-orders of correlation. Second, logical computation, as implemented by universal logic gates, primarily operates above pairwise correlation. And so, finally, there is substantial motivation to move beyond measures of pairwise correlation. We must learn to recognize hidden structures and to use higher-order structural investigations to better understand information processing. This is critical to empirically probing functionality in biological and engineered processes.

VIII Conclusion

Surprisingly, many questions we ask about structured stochastic, nonlinear processes implicate a linear dynamic over an appropriate hidden state space. That is, there is an implied hidden Markov model. The promise is that once the dynamic is found for the question of interest, one can make progress in analyzing it. Unfortunately, a roadblock immediately arises: these hidden linear dynamics are generically nondiagonalizable for questions related to prediction and to information and complexity measures. Deploying Part I’s meromorphic functional calculus, though, circumvents the roadblock. Using it, we determined closed-form expressions for a very wide range of information and complexity measures. Often, these expressions turned out to be direct functions of the HMM’s transition dynamic.

This allowed us to catalog in detail the range of possible convergence behaviors for correlation and myopic uncertainty. We then considered complexity measures that accumulate during the transient relaxation to observer synchronization. We also introduced the new notion of complexity spectra, gave a new kind of information-theoretic signal analysis in terms of coronal spectrograms, and highlighted common simplifications for special cases, such as almost diagonalizable dynamics. We closed by analyzing several families of finite and infinite Markov and cryptic order processes and emphasized the importance of higher-than-pairwise-order correlations, showing how the excess entropy spectrum is the key diagnostic tool for them.

The analytical completeness might suggest that we have reached an end. Partly, but the truth we seek is rather farther down the road. The meromorphic functional calculus of nondiagonalizable operators merely sets the stage for the next challenges—to develop complexity measures and structural decompositions for infinite-state and infinite excess entropy processes. Hopefully, the new toolset will help us scale the hierarchies of truly complex processes outlined in Refs. [5, 4, 1], at a minimum giving exact answers at each stage of a convergent series of finite-$\epsilon$-machine approximations.

Acknowledgments

The authors thank Alec Boyd, Chris Ellison, Ryan James, John Mahoney, and Dowman Varn for helpful discussions. JPC thanks the Santa Fe Institute for its hospitality. This material is based upon work supported by, or in part by, the U. S. Army Research Laboratory and the U. S. Army Research Office under contract numbers W911NF-12-1-0234, W911NF-13-1-0340, and W911NF-13-1-0390.

Appendix A Example Analytical Calculations

To exercise the operational nature of the framework introduced, the following explicitly carries out the analytic calculations to obtain the closed-form complexity measures for the $(p,q)$-parametrized $(2\text{-}1)$-GP-$(2)$ Process. This process was already visually explored in the second column of Table 2. And so, the goal here is primarily pedagogical—providing insight and better explicating particular calculational steps. The appendix demonstrates a variety of techniques in the spirit of a tutorial, though many were not called out in the main development.

A.1 Process and spectra features

The $(p,q)$-parametrized $(2\text{-}1)$-GP-$(2)$ Process is described by its $\epsilon$-machine, whose state-transition diagram was shown in the first row and second column of Table 2 and is reproduced here in Fig. 10. Formally, the $(2\text{-}1)$-GP-$(2)$ stationary stochastic process is generated by the HMM \mathcal{M}_{\epsilon\text{M}}=\bigl{(}\bm{\mathcal{S}},\mathcal{A},\{T^{({x})}\}_{{x}\in\mathcal{A}},\eta_{0}=\pi\bigr{)}. That is, $\mathcal{M}$ consists of a set of hidden causal states \bm{\mathcal{S}}=\{{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\text{A}},{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\text{B}},{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\text{C}},{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\text{D}}\}, an alphabet \mathcal{A}=\{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0},{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1},{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}2}\} of symbols emitted to form the observed process, and a set \bigl{\{}T^{({x})}:T^{({x})}_{s,s^{\prime}}=\Pr(X_{t}={x},\mathcal{S}_{t+1}=s^{\prime}|\mathcal{S}_{t}=s)\bigr{\}}_{{x}\in\mathcal{A}} of symbol-labeled transition matrices. These are:

[TABLE]

The symbol-labeled transition matrices sum to the row-stochastic internal state-to-state transition matrix:

[TABLE]

This is a Markov chain over the hidden states. From it we find the stationary state distribution $\bra{\pi}=\bra{\pi}T$:

[TABLE]

The first analysis task is to determine the eigenvalues and associated projection operators for the internal state-to-state transition matrix $T$. From:

[TABLE]

we find the four eigenvalues:

[TABLE]

All eigenvalues are real for $p\geq 2\sqrt{q}-q$. Two are complex with nonzero imaginary part when $p<2\sqrt{q}-q$. Putting this together with the transition-probability consistency constraint that $p+q<1$ yields the map of the transition-parameter space shown in Fig. 11.

For a generic choice of the parameter setting $(p,q)$, all $T$’s eigenvalues are unique, and so $T$ is diagonalizable. However, two of the eigenvalues become degenerate along the parameter subspace $p=2\sqrt{q}-q$, giving $\Lambda_{T}=\{1,0,-\sqrt{q}\}$. We find that the algebraic multiplicity $a_{-\sqrt{q}}=2$ is larger than the geometric multiplicity $g_{-\sqrt{q}}=1$, yielding nondiagonalizability ($\nu_{-\sqrt{q}}=2$) along this ($p=2\sqrt{q}-q$)-submanifold. More broadly, experience has shown that eigenvalues generically induce nondiagonalizability when they collide and scatter in the complex plane. For example, this occurs when a pair of eigenvalues first “entangle” to become complex conjugate pairs.

For any parameter setting ($p,q$), we find $T$’s right eigenvectors from $(T-\lambda I)\ket{\lambda}=\ket{0}$. They are:

[TABLE]

for all $\lambda\in\Lambda_{T}\setminus\{0\}$. Similarly, we find the left eigenvectors of $T$ from $\bra{\lambda}(T-\lambda I)=\ket{0}$:

[TABLE]

for all $\lambda\in\Lambda_{T}\setminus\{0\}$. Clearly, the left eigenvectors are not simply the complex-conjugate transpose of the right eigenvectors. This is a signature of the more intricate algebraic structure in these processes.

Since $T$ is generically diagonalizable, all of the projection operators are simply the normalized ket-bra outer products:

[TABLE]

so long as $p\neq 2\sqrt{q}-q$. To wit:

[TABLE]

Along the nondiagonalizable ($p=2\sqrt{q}-q$)-subspace of parameter settings: $\braket{{}^{-}\sqrt{q}}{{}^{-}\sqrt{q}}=0$. This corresponds to the fact that the right and left eigenvectors are now dual to the left and right generalized eigenvectors, rather than being dual to each other. We find the projection operator for the nondiagonalizable eigenspace is:

[TABLE]

where:

[TABLE]

Equivalently, $T_{-\sqrt{q}}=I-T_{1}-T_{0}$. The projection operators for the remaining ($\lambda=0,1$) eigenspaces retain the same form as before: $T_{\lambda}=\ket{\lambda}\bra{\lambda}/\braket{\lambda}{\lambda}$.

A.2 Observed correlation

The pieces are now in place to calculate the observable correlation and power spectrum. Recall that we derived the general spectral decomposition of the autocorrelation function $\gamma(L)=\left\langle\overline{{X}}_{t}{X}_{t+L}\right\rangle_{t}$:

[TABLE]

for nonzero integer $L$, where:

[TABLE]

and:

[TABLE]

For generic parameter settings, this reduces to:

[TABLE]

Notably, the autocorrelation splits into an ephemeral part (via the Kronecker delta) due to the zero eigenvalue, an exponentially decaying oscillatory part due to the two eigenvalues with magnitude between [math] and $1$, and an asymptotic part that survives even as $L\to\infty$ due to the eigenvalue of unity. Moreover, for $p<2\sqrt{q}-q$, the two nontrivial eigenvalues become complex conjugate pairs. This allows us to rewrite the autocorrelation for the $(2\text{-})$-GP-$(2)$ process for $|L|\geq 2$ concisely as:

[TABLE]

where:

[TABLE]

The latter form reveals that the magnitude of the largest nonunity eigenvalue $|\xi|=\sqrt{q}$ controls the slowest rate of decay of observed correlation $|\gamma(L)|\sim\sqrt{q}^{L}$ and that the complex phase of the eigenvalues determine the oscillations within this exponentially decaying envelope.

We found the general spectral decomposition of the continuous part of the power spectrum is:

[TABLE]

Also, recall from earlier that the delta functions of the power spectrum arise from $T$’s eigenvalues that lie on the unit circle:

[TABLE]

where $\omega_{\lambda}$ is related to $\lambda$ by $\lambda=e^{i\omega_{\lambda}}$. As long as $p+q<1$, $\lambda=1$ is the only eigenvalue that lies on the unit circle, so that:

[TABLE]

with:

[TABLE]

Putting this all together we find the complete power spectrum for the $(2\text{-}1)$-GP-$(2)$ Process:

[TABLE]

Thus, the process exhibits delta functions from the eigenvalue on the unit circle, continuous Lorentzian-like line profiles emanating from finite eigenvalues inside the unit circle which express the process’ chaotic nature, and the unique sinusoidal contribution that can only come from zero eigenvalues.

Whenever $p<2\sqrt{q}-q$, the two nontrivial eigenvalues become complex conjugate pairs. This allows us to rewrite the process’ power spectrum in a more transparent way:

[TABLE]

which is clearly symmetric about $\omega=0$.

The level of completeness achieved is notable: we calculated these properties exactly in closed form for this infinite-Markov order stochastic process over the full range of possible parameter settings. Moreover, once it constructs the process’ $\mathscr{S}$-MSP, the next section goes on to produce the same level of analytical completeness but for predictability.

A.3 Predictability

To analyze the process’ predictability, we need the $\mathscr{S}$-MSP of any of its generators. Since we started already with the $\epsilon$-machine, we directly determine its $\mathscr{S}$-MSP. This gives the mixed-state transition matrix $W=\mathcal{W}$ that, in turn, suffices for calculating both predictability in this section and the synchronization necessary for prediction in the next.

We construct the $\mathscr{S}$-MSP by calculating all mixed states that can be induced by observation from the start distribution $\pi$ and then calculating the transition probabilities between them. There are eight such mixed-state distributions over $\bm{\mathcal{S}}$. However, four of them ({\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\delta_{\text{A}}}, {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\delta_{\text{B}}}, {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\delta_{\text{C}}}, and {\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\delta_{\text{D}}}) correspond to completely synchronized peaked distributions. The other four states are new (relative to the recurrent states) transient states and correspond to transient states of recurrent-state uncertainty during synchronization. Calculating, we find the eight unique mixed-state distributions iteratively from:

[TABLE]

starting with:

[TABLE]

are:

[TABLE]

In this, the transient mixed states $\eta^{w}$ are labeled according to the shortest word $w$ that induces them. These distributions constitute the set of mixed-states $\bm{\mathcal{R}}_{\pi}$ of the $\mathscr{S}$-MSP. Moreover, each transition probability from mixed state $\bra{\eta^{w}}$ to mixed state $\bra{\eta^{wx}}$ is calculated as: $\bra{\eta^{w}}T^{(x)}\ket{\mathbf{1}}$. Altogether, these calculations yield the $\mathscr{S}$-MSP of the $(2\text{-}1)$-GP-$(2)$ Process, reproduced in Fig. 12 from Table 2 for convenience.

As a HMM, the $\epsilon$-machine’s $\mathscr{S}$-MSP is specified by the $4$-tuple: \mathcal{M}_{\text{{\mathscr{S}-MSP}}}=\bigl{(}\bm{\mathcal{R}}_{\pi},\,\mathcal{A},\,\{\mathcal{W}^{({x})}\}_{{x}\in\mathcal{A}},\,\mu_{0}\!=\!\delta_{\pi}\bigr{)}, where $\bm{\mathcal{R}}_{\pi}$ is the set of mixed states just quoted, $\mathcal{A}$ is the same observable alphabet as before, \bigl{\{}\mathcal{W}^{({x})}\bigr{\}}_{{x}\in\mathcal{A}} is the set of symbol-labeled transition matrices among the mixed states, and $\bra{\delta_{\pi}}=\begin{bmatrix}1&0&0&0&0&0&0&0\end{bmatrix}$ is the start distribution over the mixed states.

With this new linear metadynamic in hand, our next step is to calculate the eigenvalues and projection operators of the internal mixed-state-to-state transition dynamic $\mathcal{W}=\sum_{{x}\in\mathcal{A}}\mathcal{W}^{({x})}$. $\mathcal{W}$ can be explicitly represented in the block-matrix form:

[TABLE]

where:

[TABLE]

and $T$ is the same as the state-to-state internal transition matrix of the $\epsilon$-machine from earlier.

$\mathcal{W}$’s eigenvalues are thus relatively straightforward to calculate since $\text{det}(\mathcal{W}-\lambda I)=\text{det}(A-\lambda I)\text{det}(T-\lambda I)$ implies that:

[TABLE]

The new eigenvalues introduced by the feedback matrix $A$ are $\Lambda_{A}=\{0,\pm\sqrt{p}\}$ with $\nu_{0}=2$. While the other eigenvalues ($\pm\sqrt{p}$), found most easily from Part I’s cyclic eigenvalue rule, are associated with diagonalizable subspaces. It is important to note that, while $T$ was only nondiagonalizable along a very special submanifold in parameter space, the mixed-state-to-state metadynamic is generically nondiagonalizable over all parameter settings. This nondiagonalizability corresponds to a special kind of symmetry breaking of uncertainty during synchronization.

$\mathcal{W}$’s eigenvectors are most easily found through a two-step process. Specifically, $\ket{{}^{\pm}\sqrt{p}_{A}}$ and $\bra{{}^{\pm}\sqrt{p}_{A}}$ (the solutions of $A\ket{{}^{\pm}\sqrt{p}_{A}}=\pm\sqrt{p}\ket{{}^{\pm}\sqrt{p}_{A}}$ and $\bra{{}^{\pm}\sqrt{p}_{A}}A=\pm\sqrt{p}\bra{{}^{\pm}\sqrt{p}_{A}}$) are found first, and the result is used to reduce the number of unknowns when solving the full eigenequations ($\mathcal{W}\ket{{}^{\pm}\sqrt{p}_{\mathcal{W}}}=\pm\sqrt{p}\ket{{}^{\pm}\sqrt{p}_{\mathcal{W}}}$) for $\ket{{}^{\pm}\sqrt{p}_{\mathcal{W}}}$ and $\bra{{}^{\pm}\sqrt{p}_{\mathcal{W}}}$. Similarly, we can recycle the restricted eigenvectors $\ket{\lambda_{T}}$ and $\bra{\lambda_{T}}$ found earlier for the $\epsilon$-machine to reduce the number of unknowns when solving the more general eigenvector problems for $\ket{\lambda_{\mathcal{W}}}$ and $\bra{\lambda_{\mathcal{W}}}$ in cases where $\lambda\in\Lambda_{T}$. Performing such a calculation, we find:

[TABLE]

for $\lambda\in\Lambda_{T}\setminus\{0,1\}$. Moreover, $\mathcal{W}$’s eigenvectors and generalized eigenvector corresponding to eigenvalue [math] are:

[TABLE]

Above, we used the notation $\ket{0_{k}^{(m)}}$ for indexing generalized eigenvectors introduced in Part I.

All nondegenerate eigenvalues have projection operators of the form:

[TABLE]

However, the degenerate and nondiagonalizable subspace associated with the zero eigenvalue has the composite projection operator:

[TABLE]

The fact that $\bra{\delta_{\pi}}\lambda_{\mathcal{W}}\rangle=0$ for all \lambda\in\Lambda_{T}\setminus\bigl{(}\{1\}\cup\Lambda_{A}\bigr{)} is an instantiation of a general result that greatly simplifies the calculations relating to predictability and prediction.

The remaining piece for analyzing predictability is the vector of transition-entropies $\ket{\operatorname{H}(W^{\mathcal{A}})}$. A simple calculation, utilizing the fact that:

[TABLE]

when $\sum_{i}n_{i}=d$, yields:

[TABLE]

where $\log$ is understood to be the base-2 logarithm $\log_{2}$.

Putting this all together, we can now calculate in full detail the myopic entropy rate $h_{\mu}(L)$ that results from modeling the infinite-order $(2\text{-}1)$-GP-$(2)$ process as an order-$(L\text{-}1)$ Markov process:

[TABLE]

This oscillates under an exponentially decaying envelope as it approaches its asymptotic value of:

[TABLE]

Simplifying the terms in the myopic entropy rate yields:

[TABLE]

and:

[TABLE]

for the two ephemeral contributions.

For $L\geq 3$, we find for odd $L$:

[TABLE]

and for even $L$:

[TABLE]

This highlights the period-2 nature of the asymptotic decay.

The total mutual information between the observable past and observable future, the excess entropy, is:

[TABLE]

This is the total future information that can possibly be predicted using past observations. The structure of how this information is unraveled over time is revealed in the excess entropy spectrum:

[TABLE]

From this, we observe that ${\bf E}=\lim_{\omega\to 0}\mathcal{E}(\omega)$.

A.4 Synchronizing to predict optimally

To analyze the information-processing cost of synchronizing to a process, we need its $\epsilon$-machine $\mathscr{S}$-MSP. We already constructed this in the last section. Hence, we can immediately evaluate the resources necessary for synchronizing to and predicting the $(2\text{-}1)$-GP-$(2)$ Process.

The only novel piece needed for this is the vector of mixed-state entropies $\ket{\operatorname{H}[\eta]}$. A simple calculation yields:

[TABLE]

where $\log$ is again understood to be the base-2 logarithm $\log_{2}$.

An observer, tasked with predicting the future as well as possible, must synchronize to the causal state of the dynamic. During the metadynamics of synchronization, the observer on average will pick up synchronization information according to the remaining causal-state uncertainty $\mathcal{H}^{+}(L)$ after an observation interval of $L$ steps:

[TABLE]

More explicitly, for $L\geq 2$, this becomes:

[TABLE]

The total synchronization information accumulated is then:

[TABLE]

Even after synchronization, an observer must update an average of $b_{\mu}$ of its bits of information per observation and must keep track of a net $C_{\mu}$ bits of information to stay synchronized, where:

[TABLE]

An interesting feature of prediction is a process’ crypticity:

[TABLE]

This gives, as a function of $p$ and $q$, the minimal overhead of additional memory about the past—beyond the information that the future can possibly share with it—that must be stored for optimal prediction.

In summary, we now more fully appreciate, via this rather complete analysis, the fundamental limits on predictability of our example stochastic process. It showed many of the qualitative features, both in terms of the calculation and system behavior, that should be expected when analyzing prediction, based on the more general results of the main development. The procedures can be commandeered to apply to any inference algorithm that yields a generative model, whether classical machine learning, Bayesian Structural Inference [10], or any other favorite inference tool. With a generative model in hand, synchronizing to real world data—necessary to make good predictions about the real world’s future—follows the $\mathscr{S}$-MSP metadynamics. The consequences for prediction is that typically there will be a finite epoch of symmetry collapse followed by a slower asymptotic synchronization that allows improved prediction, as longer observations induce a refined knowledge of what lies hidden.