The behaviour of information flow near criticality

TL;DR

This study uses stochastic simulations of a 2D Ising model to analyze how information transmission varies near criticality, revealing optimal conditions for information flow depending on temperature and input dynamics.

Contribution

It provides new insights into how information transmission depends on system parameters near criticality using detailed stochastic simulations.

Findings

Mutual information varies non-monotonically with temperature.

Maximum information transmission rate occurs at an optimal input correlation time.

Optimal temperature for information transfer is above the critical point and shifts with distance.

Abstract

Recent experiments have indicated that many biological systems self-organise near their critical point, which hints at a common design principle. While it has been suggested that information transmission is optimized near the critical point, it remains unclear how information transmission depends on the dynamics of the input signal, the distance over which the information needs to be transmitted, and the distance to the critical point. Here we employ stochastic simulations of a driven 2D Ising system and study the instantaneous mutual information and the information transmission rate between a driven input spin and an output spin. The instantaneous mutual information varies non-monotonically with the temperature, but increases monotonically with the correlation time of the input signal. In contrast, the information transmission rate exhibits a maximum as a function of the input correlation time. Moreover, there exists an optimal temperature that maximizes this maximum information transmission rate. It arises from a tradeoff between the necessity to respond fast to changes in the input so that more information per unit amount of time can be transmitted, and the need to respond to reliably. The optimal temperature lies above the critical point, but moves towards it as the distance between the input and output spin is increased.

Equations

$$I_{\mathrm{inst}}(S;X)=H(S)-H(S|X),$$

$$I_{\text{R}}=\frac{1}{L}\underset{L\to \infty }{\lim }I({\mathbf{S}}_L;{\mathbf{X}}_L),$$

$$I({\mathbf{S}}_L;{\mathbf{X}}_L)=H({\mathbf{S}}_L)+H({\mathbf{X}}_L)-H({\mathbf{S}}_L,{\mathbf{X}}_L)$$

$$I_{\text{R}}(\Delta t\to \delta t)=\frac{1}{L}\underset{L\to \infty }{\lim }I({\mathbf{S}}_L;{\mathbf{X}}_L,\Delta t),$$

$$I({\mathbf{S}}_L;{\mathbf{X}}_L,\Delta t)=H({\mathbf{S}}_L,\Delta t)+H({\mathbf{X}}_L,\Delta t)-H({\mathbf{S}}_L,{\mathbf{X}}_L,\Delta t),$$

$$I_{\text{R}}(\Delta t\to \delta t)=\underset{L\to \infty }{\lim }\frac{I({\mathbf{S}}_L;{\mathbf{X}}_L,\Delta t)-I({\mathbf{S}}_{L-\Delta t};{\mathbf{X}}_{L-\Delta t},\Delta t)}{\Delta t}.$$

$$P(S_t=1|S_{t-1}=-1)=P(S_t=-1|S_{t-1}=1)=r$$

$$P(S_t=1|S_{t-1}=1)=P(S_t=-1|S_{t-1}=-1)=(1-r).$$

$$h(\mathcal{S})=H(S_t|S_{t-1})=-r\log r-(1-r)\log (1-r),$$

$$H({\text{S}}_L)$$

$$\displaystyle=\log(2)-\frac{L}{\delta t}\big{[}r\log r+(1-r)\log(1-r)\big{]}.$$

$$P(S_t|S_{t-\Delta t})=\{\begin{array}{ll}{\sum }_{i=0,\mathrm{even}}^{\Delta t}(1-r{)}^{\Delta t-i}{r}^i\frac{\Delta t!}{i!(\Delta t-i)!}=q(\Delta t),& \text{for}{S}_t=S_{t-\Delta t}\\ {\sum }_{i=1,\mathrm{odd}}^{\Delta t}(1-r{)}^{\Delta t-i}{r}^i\frac{\Delta t!}{i!(\Delta t-i)!}=(1-q(\Delta t)),& \text{for}{S}_t=-S_{t-\Delta t},\end{array}$$

$$h(\mathcal{S},\Delta t)=H(S_t|S_{t-\Delta t})$$

$$=q(\Delta t)\log (q(\Delta t))+(1-q(\Delta t))\log (1-q(\Delta t)),$$

$$H(\mathbf{S}_{L},\Delta t)=\log(2)-\frac{L}{\Delta t}\big{[}q(\Delta t)\log(q(\Delta t))+(1-q(\Delta t))\log(1-q(\Delta t))\big{]}.$$

Full text

Most, if not all, living organisms need to respond to changes in their environment. Examples include bacteria searching for food, animals trying to catch prey, or birds in flocks trying to coordinate their motion. In all these cases, the flow of information, be it via an intracellular biochemical network, an intercellular neural network, or between the individuals within the group, is vital to function. Moreover, in all these examples not only the reliability of information transmission is important, but also the speed and the distance over which the information needs to be transmitted.

Recent experiments indicate that many biological systems self-organise at their critical point. Examples are the flocking behaviour of starlings Cavagna et al. (2010), signal percolation within a bacterial community Larkin et al. (2018), neural networks Tkacik et al. (2015), and cooperative transport in ants Feinerman et al. (2018). The critical point is a special point in the phase diagram, right at the border between order and disorder and characterized by the divergence of thermodynamic variables Chandler (1987). The observation that different systems self-organize near the critical point hints at a common design principle Mora and Bialek (2011). In this letter, we investigate whether criticality enhances the speed and reliability of information transmission.

The observation that very different systems self-organize near the critical point makes the Ising system, because of its generic properties, a natural choice to study information flow. Different measures have been employed to characterize information transmission in the Ising system. One is the average mutual information Shannon (1948) between pairs of neighboring spins Matsuda et al. (1996); Gu et al. (2008). It quantifies the correlations between these spins and can thus be interpreted as a measure for the reliability of information transmission. It exhibits a sharp peak at the critical point Matsuda et al. (1996); Gu et al. (2008). However, the mutual information does not contain dynamical aspects of information flow. An information theoretic measure that does include this is the transfer entropy Schreiber (2000). Barnett et al. showed that the transfer entropy between pairs of neighboring spins peaks at the critical point while a global transfer entropy measure peaks above it Barnett et al. (2013). However, the authors studied systems in thermodynamic equilibrium, while biological systems, if not any information-processing device, are driven out of equilibrium via the driving of the input signal. Moreover, they only studied the one-step transfer entropy, which does not take into account that the dynamics of the output may depend on the history of the input.

In this Letter, we investigate the flow of information in a driven 2D Ising system. The input signal $\mathcal{S}$ is a spin, the input spin, which is flipped according to a stationary random telegraph process with a time scale $\tau_{\rm s}$. Since this driving signal is stationary, our system is in a stationary non-equilibrium steady-state. The output signal $\mathcal{X}$ is another spin, the output spin, which is located at a distance $d$ away from the input spin, see Fig. 1. The input and output signals produce the random variables $S,X\in\{+1,-1\}$ at each point in time, respectively. Because the information is propagated from the input to the output spin via the other spins, the dynamics of the output are distinctly non-Markovian. Consequently, we need to recognize the history of the input and output signal in characterizing the information flow between them.

To characterize information transmission, we study two measures: the instantaneous mutual information $I_{\rm inst}$ and the information transmission rate $I_{\rm R}$. The measure $I_{\rm inst}(S;X)$ is the mutual information between the stationary input and output signal at a single point in time:

[TABLE]

where $H(S)$ is the Shannon entropy of the input signal and $H(S|X)$ is the Shannon entropy conditional on the output signal. The instantaneous mutual information has been used to study information transmission in intracellular signaling networks Tostevin and Ten Wolde (2010); Brittain et al. (2017); Das et al. (2017); Malaguti and ten Wolde (2019). In contrast to the mutual information studied by Matsuda et al. Matsuda et al. (1996), which characterizes equilibrium correlations between pairs of spins, $I_{\rm inst}(S;X)$ quantifies the non-equilibrium correlations between the input and output spin of our driven system. These correlations depend on the input time scale $\tau_{\rm s}$ and the response time $\tau_{\rm r}$ of the system, which is determined by the temperature.

While the instantaneous mutual information $I_{\rm inst}(S;X)$ quantifies how accurately the input spin is mapped onto the output spin, it does not quantify the rate of information transmission. The latter is not only determined by the accuracy of the input-output mapping, but also by the rate at which independent “messages” are transmitted through the system. Autocorrelations within the input and the output signal lower the information transmission rate. To take these correlations into account, we study the information transmission rate, which is defined as the rate at which the mutual information between the trajectories of the input and output signal increases Tostevin and Ten Wolde (2009):

[TABLE]

where $\mathbf{S}_{L}=[S(t_{1}),S(t_{2}),\dots,S(t_{n})]$ and $\mathbf{X}_{L}=[X(t_{1}),X(t_{2}),\dots,X(t_{n})]$ are spin trajectories of duration $L=(n-1)\delta t$, containing $n$ subsequent spin states $S$ ($X$) at successive time points $t_{i}=(i-1)\delta t$, with $\delta t$ the elementary timestep of the dynamics SI . To capture the autocorrelations in the input and output signal, the trajectory lengths have to be longer than the longest timescale in the problem, $L>\tau_{\rm s},\tau_{\rm r}$; $I_{\rm R}$ then properly takes into account the history of the input and output spin, in contrast to the one-step transfer entropy. We note that if there were no autocorrelations in the input and output signal, the information transmission rate would reduce to $I_{\rm R}=I_{\rm inst}/\delta t$. However, in general, $I_{\rm R}$ is lower than $I_{\rm inst}/\delta t$, precisely because of the signal autocorrelations. We also point out that since the output signal does not feed back on the input, the information transmission rate can be related to the multi-step transfer entropy Massey (1990).

In order to evaluate the effects of the dynamics and criticality on information flow, we will study both measures as a function of the time scale of the input signal $\tau_{\rm s}$ for different temperatures $T$ close to the critical temperature $T_{\rm c}$ and for different distances $d$ between the input and output signal. We are mainly interested in temperatures higher than the critical temperature, since for lower temperatures the system freezes down in the ferromagnetic phase, drastically slowing down information transmission. We will show that the non-trivial interaction between the diverging correlation length and the diverging response time near the critical point causes the information flow to be optimal close to, but not at, the critical point. The optimal temperature is determined by the distance over which the information needs to be transmitted and the size of the system.

Consider a 2D Ising system of ${\mathcal{N}}\times{\mathcal{N}}$ spins with periodic boundary conditions and no external magnetic field. For a spin configuration $\bm{\sigma}=\sigma_{1},\dots,\sigma_{\mathcal{N}}\in\{+1,-1\}$, the Hamiltonian of the system is $H(\bm{\sigma})=-J\sum_{\langle i,j\rangle}\sigma_{i}\sigma_{j}$, where $J$ is the coupling parameter and the sum is taken over all nearest neighbours. For isotropic coupling, the critical temperature is $k_{\rm B}T_{\rm c}/J=2.269$ Onsager (1944). Following Barnett et al. Barnett et al. (2013), we use discrete-time Glauber spin-flip dynamics Glauber (1963). We define the response time $\tau_{\rm r}$ as the time scale over which spontaneous fluctuations in the undriven system, as computed via the two-point time correlation function of the input and output spin, relax to equilibrium Chandler (1987). Entropies are measured in nats.

The information transmission rate is notoriously difficult to compute, because the state space of the input and output trajectories rapidly diverges with the length of the trajectories. We have therefore considered not only relatively small systems, but also developed the following scheme: To limit the size of the state space, we introduce a sampling interval $\Delta t$ such that the trajectory length $L=(n-1)\Delta t$, where $n$ is the number of spin states in both the input and output trajectory. As described in SI , we verify that $L$ is longer than the input and output correlation time such that $I(\mathbf{S}_{L};\mathbf{X}_{L})$ increases linearly with $L$ and the information transmission rate $I_{\rm R}$ is independent of $L$. We then compute for long enough $L$, $I_{\rm R}=I_{\rm R}(\Delta t$) for a range of $\Delta t$ values, where we verify that the entropy histograms are sampled accurately, using the Bayesian entropy estimator of Nemenman et al. to enhance the estimate of the (joint) entropies Nemenman and Bialek and De Ruyter van Steveninck (2004). We then extrapolate $I_{\rm R}(\Delta t)$ to the quantify of interest, $I_{\rm R}(\Delta t\to\delta t)$, where $\delta t$ is the elementary time step of the Glauber dynamics; to verify this extrapolation procedure, we have also recomputed by simulations $I_{\rm R}(\Delta t)$ for a number of extrapolated $\Delta t$ values (see SI ).

Figure 2 shows the instantaneous mutual information $I_{\rm inst}(S;X)$ between the input and output signal separated by a distance $d=2$ as a function of the input correlation time $\tau_{\rm s}$ and temperature $T$ in an Ising system of 5 $\times$ 5 spins. The instantaneous mutual information rises with the input correlation time $\tau_{\rm s}$ (Fig. 2a), because this gives the system more time to respond to changes in the input signal and hence more time to correlate the output with the input signal. For large $\tau_{\rm s}$, the instantaneous mutual information reaches a plateau value $I_{{\rm inst},\infty}(S;X)$ that corresponds to the static mutual information, which is the mutual information between the output spin and the input spin when the latter is held fixed indefinitely for each realization $S=1,-1$. The static mutual information increases as the temperature is decreased, because decreasing the temperature lowers the thermal noise in the transmitted signal.

Panel b of Fig. 2 shows that for a given correlation time $\tau_{\rm s}$ of the input signal, there exists an optimal temperature $T_{\rm opt}$ that maximises the instantaneous mutual information $I_{\rm inst}(S;X)$. Increasing the temperature raises the thermal noise in the signal, which tends to lower the instantaneous mutual information. On the other hand, increasing the temperature also reduces the response time $\tau_{\rm r}$. This allowes the system to more accurately track the input signal, which tends to raise the instantaneous mutual information between the input and output signal. The interplay between these two effects gives rise to an optimal temperature $T_{\rm opt}(\tau_{\rm s})$ that maximizes the instantaneous mutual information, $I_{\rm inst}^{\rm max}(\tau_{\rm s})$. This optimal temperature decreases as the input correlation time $\tau_{\rm s}$ is increased, because this gives the system more time to respond to the variations in the input. Moreover, the maximum instantaneous mutual information $I_{\rm inst}^{\rm max}(\tau_{\text{s}})$ rises with $\tau_{\rm s}$, not only because increasing $\tau_{\rm s}$ raises $I_{\rm inst}$ by itself, but also because the lower optimal temperature $T_{\rm opt}(\tau_{\rm s})$ reduces the thermal noise in the signal.

Figure 3a shows the information transmission rate $I_{\rm R}$ as a function of the correlation time of the input signal $\tau_{\rm s}$ for different temperatures $T$. While, for a given temperature, the instantaneous mutual information $I_{\rm inst}$ increases monotonically with the input correlation time $\tau_{\rm s}$ (see Fig. 2a), the information transmission rate $I_{\rm R}$ exhibits an optimal $\tau_{\rm s}$ that maximizes the information transmission rate. When $\tau_{\rm s}$ is too short, the signal is changing faster than the output can respond to, which tends to decrease $I_{\rm R}$ by increasing the conditional entropy $H(\textbf{S}_{L}|\textbf{X}_{L})$ (see Eq. 2). On the other hand, for large $\tau_{\rm s}$ time is wasted when the output has been correlated to the input yet is waiting for the signal to change again; indeed, the entropy of the input signal $H(\textbf{S}_{L})$ rises as $\tau_{\rm s}$ is decreased, which tends to enhance $I_{\rm R}$ (see Eq. 2). This interplay causes the information transmission rate to have a maximum at an optimal input time scale $\tau_{\rm s}^{\rm opt}$. The value of $\tau_{\rm s}^{\rm opt}$ decreases with temperature, because at higher temperatures the system can respond more rapidly to changes in the input signal.

Panel b of Fig. 3 shows the maximum value of the information transmission rate $I_{\rm R}$ at the optimal input correlation time $\tau_{\rm s}$, $I_{\rm R,max}$, as a function of the temperature $T$. Clearly, there exists an optimal temperature that maximizes $I_{\rm R,max}$. This is in marked contrast to the maximum value of the instantaneous mutual information, obtained for $\tau_{\rm s}\to\infty$, which increases monotonically with decreasing temperature, even for temperatures below $T_{\rm c}$ (see Fig. 2a). The optimum in $I_{\rm R,max}$ arises from the trade-off between a faster response at higher temperatures, which allows for a more rapidly varying input, thereby increasing the entropy of the input signal (see Eq. 1), and less thermal noise in the transmitted signal at lower temperatures. That $I_{\rm R,max}$ peaks above the critical temperature is because the response time rapidly increases near the critical temperature, thereby decreasing the amount of information per unit time that can be sent through the system.

So far we have kept both the distance $d$ between the input and output spin constant, as well as the system size. We now systematically vary these parameters. Figure 4 shows the maximum information transmission rate $I_{\rm R,max}$, obtained by optimizing over $\tau_{\rm s}$ (see Fig. 3) , as a function of temperature $T$ for different values of $d$ in a $10\times 10$ Ising system. The information transmission rate decreases as $d$ is increased, because the correlations between spins become weaker as the distance between them becomes larger. More interestingly, the optimal temperature that maximizes $I_{\rm R,max}$ moves closer to the critical temperature when the distance between the signal and the output is increased. When the distance between the input and output spin is increased, the correlation length must be increased in order to maintain the correlations between the input and output spin. This can be achieved by bringing the system closer to the critical point.

Critical effects are stronger in systems of larger size. Close to $T_{\rm c}$, the response time of our system increases up to six-fold when the system size is increased from $5\times 5$ to $10\times 10$ spins. This makes it beneficial for information transmission to move the system further away from the critical point when the system size is increased at constant $d$. Compare the case of $d=2$ in the $5\times 5$ system in Fig. 3 with $d=2$ in the $10\times 10$ system in Fig. 4: the optimal temperature shifts from ${T}_{\rm opt}\approx 2.5$ to $2.6$ in the larger system. The maximum information transmission rate $I_{\rm R,max}$ decreases because of the larger response time.

As the system is moved closer to the critical temperature, both the correlation length and the correlation time increase, which have opposite effects on information transmission. Moreover, these effects increase with the system size, diverging in the thermodynamic limit. Hence, when $d$ is increased in a system of constant size, the optimal temperature $T_{\rm opt}$ that maximizes information transmission moves towards $T_{\rm c}$, because the increasing correlation length is necessary to transmit information over large distances, as Fig. 4 shows. It also means that when the system size is increased at constant $d$, $T_{\rm opt}$ moves away from $T_{\rm c}$, because of the increasing response time, as discussed above. This raises the question how $T_{\rm opt}$ changes as $d$ is scaled together with the system size, which, as renormalization group theory indicates, is also the relevant finite-size scaling question for this problem. We have therefore also performed simulations for $d=6$ and ${\cal N}=15$. The optimal temperature that maximizes information transmission decreases from $T_{\rm opt}\approx 2.53$ for ($d=2$, ${\cal N}=5$), to $T_{\rm opt}\approx 2.44$ for ($d=4$, ${\cal N}=10$), and $T_{\rm opt}\approx 2.38$ for ($d=6$, ${\cal N}=15$) (see Fig. S9 SI ). Our results thus suggest that $T_{\rm opt}$ moves towards $T_{\rm c}$ in the thermodynamic limit.

In summary, the information transmission rate is a dynamic quantity that is influenced by both the strength of the correlation that can be achieved between the input and the output signal, and the time scale on which the output can respond to a change in the input signal. These two properties are oppositely influenced by the temperature of the system. The system faces a trade-off between increasing the temperature in order to decrease the response time such that more information per unit time can be transmitted through the system and decreasing the temperature in order to increase the correlations between the input signal and output. This trade-off produces a maximum information transmission rate at an optimal temperature that depends on the distance between the in- and output spin and the size of the system. The optimal temperature is close to yet above the critical point, although our results leave open the possibility that it moves towards the critical temperature in the thermodynamic limit. Our results may explain why a number of biological systems appear to be tuned near the critical point Cavagna et al. (2010); Larkin et al. (2018); Tkacik et al. (2015); Feinerman et al. (2018), and may also be relevant for understanding information transfer in systems outside the realm of biology. Lastly, many systems, including biological systems, are higher dimensional. Since the response time does not depend on the dimensionality of the system while correlations decay faster with distance in higher dimension, we conjecture that in higher dimensional systems the optimal temperature is closer to the critical point.

This work is part of the research programme of the Netherlands Organisation for Scientific Research (NWO) and was performed at AMOLF. Ito was funded by grants KAKENHI Grant No. JP16K17780, and JST Presto Grant No. JP18070368, Japan. We thank Ilya Nemenman for sharing his code to estimate entropies and Tom Ouldridge for a careful reading of our manuscript.

Estimation of the entropy of the input signal

Because it is possible to analytically compute the entropy of the input signal, the inference of the entropy of the input signal provides a good test case for our procedure to estimate the entropy. At each elementary time step, there is a chance $r=\delta t/\tau_{\rm s}$ that the spin flips:

[TABLE]

where $S_{t}$ is the state of the spin at time $t$ (in units of $\delta t$). Similarly,

[TABLE]

The signal is a Markovian process since the chance of a spin flip does not depend on the history of the trajectory. The entropy rate of this process is then given by:

[TABLE]

where we assume that the spin up and spin down states are equally likely. Using this entropy rate, the quantity of interest, the true entropy of the input signal, for $\Delta t\to\delta t$, is

[TABLE]

We now consider the effect of sampling the input trajectory at a sampling interval $\Delta t$. The chance of a spin flip at the next time interval $\Delta t$ is

[TABLE]

where we sum over each possible step where the spin could flip within a sampling interval $\Delta t$. The entropy rate is thus

[TABLE]

such that the entropy is

[TABLE]

This expression reduces to Eq. 7, when $\Delta t/\delta t=1$, as it should.

It is possible to compare this theoretical value of $H(\mathbf{S}_{L},\Delta t)$ with estimates of the entropy using simulations of the input signal for a given sampling interval. Different estimators have been proposed to estimate the entropy Miller (1974); Grassberger (2008). Here, we compare the naive estimator, in which the probability of a specific trajectory is simply given as $P(\mathbf{S}_{L}=\mathbf{s}_{L})=N_{\mathbf{s}_{L}}/N_{\rm tot}$, where $N_{\mathbf{s}_{L}}$ is the number of observations of the trajectory $\textbf{s}_{L}$, and $N_{\rm tot}$ is the total number of observations, to the estimator proposed by Nemenman et al. Nemenman and Bialek and De Ruyter van Steveninck (2004), called the NSB estimator. When the performance of the two estimators are compared, we see, by comparing the computational estimates to the theoretical value, that the NSB estimator has overall a smaller error than the naive estimator. However, both of these estimators suffer from a bias at large $L$, where the entropy of the input signal is underestimated because the number of states of the input trajectory, $K=2^{n}$, exceeds the number of observations. We have chosen the NSB estimator, but also have developed a procedure to estimate the information transmission rate without undersampling.

For sufficient sampling, we can reliably estimate the mutual information

When we increase the length of the trajectories $L$ by increasing $n$ keeping $\Delta t$ constant, we can distinguish three regimes for the mutual information $I({\bf S}_{L};{\bf X}_{L},\Delta t)$, see Fig. S3: first, the mutual information increases with a low, constant rate. Here, we can reliably estimate the entropies of all three ensembles $\{\textbf{S}_{L}\}(\Delta t)$, $\{\textbf{X}_{L}\}(\Delta t)$, and $\{\textbf{S}_{L},\textbf{X}_{L}\}(\Delta t)$ and the slope of the mutual information equals the information transmission rate $I_{\rm R}(\Delta t)$ for this value of the sampling interval $\Delta t$. Then as $L=(n-1)\Delta t$ is increased further (by raising $n$), the mutual information rises at a higher pace. In this regime, only the entropy estimation of the joint trajectory $\{\textbf{S}_{L},\textbf{X}_{L}\}(\Delta t)$ suffers from undersampling, causing $H({\bf S}_{L};{\bf X}_{L},\Delta t)$ to be underestimated. The joint trajectory suffers from undersampling first, because it contains twice the number of spin states as compared to the input or output trajectories. Since the entropy $H({\bf S}_{L};{\bf X}_{L},\Delta t)$ of the joint trajectory is subtracted from the mutual information $I({\bf S}_{L};{\bf X}_{L},\Delta t)$, its underestimation will cause the mutual information to be overestimated (see Eq. S4). Finally, at larger values of $L$, the slope of the mutual information decreases again. All three entropies are now underestimated and the slope of the mutual information decreases. Clearly, only in the initial linear regime, the information transmission rate can be reliably inferred from the slope of the mutual information $I({\bf S}_{L};{\bf X}_{L},\Delta t)$.

By increasing the number of observations, the initial regime is valid for a larger range of trajectory lengths $L$. By increasing the number of observations with a factor of $10^{2}$, the correct regime is elongated with approximately $n\approx 5$ spin states in the trajectories. Additionally, the collapse of all three lines in the initial regime gives us confidence that we can reliably estimate the mutual information when the trajectory does not contain too many spin states. When the estimate of the mutual information does not change when we repeat the simulation with more observations, then we can be confident of our estimate of the mutual information: as we saw in Fig. S2, the NSB estimator does not have any bias when there is sufficient sampling.

From inspection of Fig. S3, we see that for $N=10^{7}$ observations, the mutual information $I(\textbf{S}_{L};\textbf{X}_{L},\Delta t)$ stays in the initial, correct, linear regime up to $L\approx 72$, which corresponds to $n\approx L/\Delta t+1=72/8+1\approx 10$ spin states in the trajectory, corresponding to a state space of $K=2^{2n}\approx 10^{6}$ for the join trajectory. For the results of the main text, we have used $N=4*10^{7}$ for $n=9$ spin states in the input and output trajectories. When increasing or decreasing the number of spin states $n$, we adjusted the number of observations $N$ according to the change in the size of the state space $K$. Using these parameters, we have $N>>K$ and there is a vanishingly small error on the estimate of the mutual information.

The information transmission rate increases for a smaller sampling interval

To reliably estimate the information transmission rate, it is necessary to compute the rate at a sufficiently long trajectory length $L$, which should be longer than the longest timescale in the system, $L>\tau_{\rm s},\tau_{\rm r}$, where $\tau_{\rm s}$ is the input timescale and $\tau_{\rm r}$ the response time of the system. Yet, the number of spin states in the trajectories, $n$, cannot be too large because this will create a sampling problem, as discussed above. We thus need to increase the sampling interval $\Delta t$ beyond $\delta t$. However, for a given overall trajectory length $L=(n-1)\Delta t$, the mutual information $I(\textbf{S}_{L};\textbf{X}_{L},\Delta t)$ depends on $\Delta t$ while we would like to obtain the limit $\Delta t\to\delta t$, which is the elementary time step of the Glauber dynamics.

Figure S4 illustrates how the information transmission rate $I_{\rm R}(\Delta t)$ increases for decreasing sampling interval $\Delta t$. The 6 black points show the computed information transmission rate $I_{\rm R}(\Delta t)$ for 6 different values of $\Delta t$. For these 6 values of $\Delta t$, the number of observations $N=5\times 10^{7}$ and the number of spins states in the trajectory $n=5$ is kept constant, such that $N\gg K$, where $K$ is the number of unique possible trajectories (see previous section), and we can reliably estimate the entropies of the trajectories. Because $n$ is constant for these 6 black points, the trajectory length $L=(n-1)\Delta t$ decreases for smaller $\Delta t$. Yet, the trajectories remain long enough, meaning that $L>\tau_{\rm s},\tau_{\rm r}$ (at the temperature $T=2.45$ of the simulations, the response time $\tau_{\rm r}=51$ and the correlation time of the input signal $\tau_{\rm s}=63$). The mutual information $I({\bf S}_{L};{\bf X}_{L},\Delta t)$ thus increases linearly with $L$. From the slope of $I({\bf S}_{L};{\bf X}_{L},\Delta t)$ as a function of $L$, i.e. from Eq. S5, we can therefore reliably estimate the information transmission rate $I_{\rm R}(\Delta t)$ for each value of $\Delta t$

To get the quantity of interest, the information transmission rate $I_{\rm R}(\Delta t\to 1)$, we fit a quadratic function to the estimates of $I_{\rm R}(\Delta t)$ at the 6 values of $\Delta t$ corresponding to the black points. This function is the black dashed line. This function is then extrapolated to $\Delta t=1$ to retrieve the value of the information transmission rate at the elementary time step of the Glauber dynamics (see the extrapolated black dashed line). In the case of Fig. S4, we find a value of $I_{\rm R}(\Delta t=1)=0.0048$.

In order to verify this scheme, we have recomputed the information transmission rate $I_{\rm R}(\Delta t)$ for a number of extrapolated values of $\Delta t$; these are the red points. For these smaller sampling intervals $\Delta t$ we have increased the number of spin states $n$ in the trajectories to ensure that the trajectory length remains long enough, ie. $L>\tau_{\rm s},\tau_{\rm r}$. At the smallest sampling interval, $\Delta t=5$, we used $n=16$ such that $L=75$. For these values of $\Delta t$ corresponding to the red points, the number of samples is increased to $N=10^{8}$ and the errorbar is estimated using the NSB method. The figure illustrates that at small $\Delta t$ the computed values (red points) overestimate the information transmission rate as compared to the extrapolated values, given by the black dashed line. In the inset, we see however that this overestimation is due to undersampling: by increasing the number of observations $N$, the computed information transmission rate decreases to approach the extrapolated value, given by the horizontal line. The information transmission rate is inversely proportional to the number of observations, as is shown by the fit in the dashed line in the inset. Comparing the value that $I_{\rm R}$ decays to, $B=0.0046$, with the extrapolated value from the main panel, $I_{\rm R}(\Delta t=5)\approx 0.0044$, shows that our extrapolation procedure gives a reliable estimate of the information transmission rate for smaller $\Delta t$ values. We also note that this figure underscores the observation of Fig. 1, namely that even the NSB method suffers from undersampling.

The sampling interval must be smaller that the input

correlation time and the response time of the system

Above we discussed that the trajectory length $L=(n-1)\Delta t$ must be larger than $\tau_{\rm s},\tau_{\rm r}$ and that $n$ cannot be too large because of undersampling. Moreover, we described how the information transmission rate of interest, i.e. $I_{\rm R}(\Delta t)$ at $\Delta t=1$, can be obtained by extrapolating $I_{\rm R}(\Delta t)$ computed for large $\Delta t$ to $\Delta t\to 1$ (see Fig. S4). However, how do we choose the sampling interval $\Delta t$ for which we compute the information transmission rate? As mentioned, it is necessary that $L=(n-1)\Delta t>\tau_{\text{s}},\tau_{\text{r}}$; given that the maximum value of $n$ that allows for good sampling in reasonable CPU time is finite $(n\approx 10)$, $\Delta t$ needs to be large enough. However, it is not possible to indefinitely increase the sampling interval: $\Delta t$ must be smaller than the correlation time of the input signal and the response time of the system. This is illustrated in Fig. S5. Both panels correspond to a $5\times 5$ spin system at a temperature $T=2.3$, in which the response time $\tau_{\rm r}=88$ and the correlation time of the input signal $\tau_{\rm s}=300$. While keeping the number of spin states constant, $n=9$, $\Delta t$ and $L=(n-1)\Delta t$ are varied, in both panels. The two panels differ in which range of $\Delta t$ values is used to extrapolate to $\Delta t=1$. In panel a, the red dots and the red dashed line correspond to a scheme in which the extrapolation is based on $\Delta t$ values that are larger than the response time $\tau_{\rm r}$ of the system. In contrast, the blue dots and blue dashed line correspond to a scheme in which the extrapolation is based on $\Delta t$ values that are all smaller than $\tau_{\rm r}$. Clearly, the extrapolation of the red scheme, based on $\Delta t$ values larger than $\tau_{\rm r}$, severely underestimates the extrapolated value of $I_{\rm R}$. We thus need to use $\Delta t$ values that are shorter than $\tau_{\rm s},\tau_{\rm r}$. This is further supported by panel b. In this panel, two extrapolation schemes are shown, which differ in the values of $\Delta t$ used for the extrapolation. In contrast to panel a, however, both of these schemes use $\Delta t$ values that are all smaller than $\tau_{\rm s},\tau_{\rm r}$. Clearly, both schemes give essentially the same extrapolated value of $I_{\rm R}$, even though the extrapolation is based on different values of $\Delta t$.

Sampling parameter requirements

In summary, the parameters of the sampling procedure must satisfy the following constraints:

$\Delta t$ must be smaller than $\tau_{\rm s},\tau_{\rm r}$ 2. 2.

yet $L=(n-1)\Delta t$ must be larger than $\tau_{\rm s},\tau_{\rm r}$. 3. 3.

$N$ must be larger than $2^{2n}$ so that undersampling does not occur.

When these three criteria are met, the extrapolation procedure illustrated in Figs. S4 and S5 yields a reliable estimate of $I_{\rm R}(\Delta t=1)$.

Pseudo-code

• •

We have computed for each temperature $T$ the correlation time $\tau_{\text{c}}$ from the decay of the two-point time correlation function $\langle S(0)X(t)\rangle$, which serves as a measure for the response time of the system, $\tau_{\text{r}}$; in fact, simulations reveal that this response time is similar to the timescale over which the total magnetisation of the system relaxes to zero when the input spin, which had been held fixed, is allowed to thermally equilibrate, indicating that the driving of the system via the flips of the input spin keeps the system in the linear-response regime Chandler (1987). In the optimal systems that maximize information transmission, $\tau_{\rm r}$ is typically on the order of the correlation time $\tau_{\rm s}$ of the input signal.

• •

For a given temperature $T$ and correlation time of the input signal $\tau_{\text{s}}$, we choose six sampling intervals $\Delta t<\tau_{\rm s},\tau_{\rm r}$.

• •

For each sampling interval $\Delta t$, we calculate $I_{\text{R}}(\Delta t)$ and $I_{\text{inst}}$. The number of observations $N$ and spin states $n$ in a trajectory depend on the temperature. At low temperatures, the long response time allows for relatively large sampling intervals such that we use $n=8$ and $N=10^{7}$, while still having trajectory lengths $L>\tau_{\rm s},\tau_{\rm r}$. At higher temperatures, the response time becomes shorter, such that the maximum sampling interval $\Delta t$ that we can use also decreases. In order to still satisfy the condition on the trajectory lengths $L$, we increase the number of spin states in a trajectory to $n=10$ at $T=2.7$, while increasing the number of observations accordingly to $N=16\times 10^{7}$, since $N>K=2^{2n}$ for estimating the joint entropy.

• •

Samples of the trajectories $\{\textbf{S}_{L}\}(\Delta t)$, $\{\textbf{X}_{L}\}(\Delta t)$, and $\{\textbf{S}_{L},\textbf{X}_{L}\}(\Delta t)$ with different sampling intervals are collected in parallel when simulating the driven Ising system. As illustrated in Fig. S1, the signal produces a new spin state every time step $\delta t$ with a correlation time of $\tau_{\rm s}$. Samples for the trajectory $\{\textbf{S}_{L}\}$, which is characterised by a number of spin states $n$ and the sampling interval $\Delta t$, are collected by storing spin states at every $\Delta t$ in a vector until it reaches length $L$. Trajectories with the same sampling interval are collected in sequence and do not overlap, while trajectories with different sampling intervals are collected in parallel and do overlap with each other. Samples of the output and joint trajectories are collected similarly.

• •

Using the Bayesian estimator of Nemenman et al. Nemenman and Bialek and De Ruyter van Steveninck (2004), we estimate the entropies of the (joint) trajectories and compute the mutual information and information transmission rate according to Eqs. 1 and 2 as a function of the sampling interval.

• •

By extrapolating the information transmission rate to $\Delta t=1$, we get the value $I_{\text{R}}(\tau_{\text{s}},T)$ that is plotted in Figs. 3 and 4 of the main text.

The information rate in a system of $10\times 10$ spins

Similar to Figs. 3 and 4 of the main text, Figs. S6 and S7 show, respectively, the instantaneous mutual information $I_{\rm inst}$ and the information transmission rate $I_{\rm R}$ as a function of the input signal correlation time $\tau_{\text{s}}$ for a larger system, of $10\times 10$ spins, and for different distances $d$. Because the effects of criticality are stronger in the larger system, the differences in response times $\tau_{\rm r}$ between different temperatures $T$ are larger. For this reason, we investigate a broader range of values for $\tau_{\text{s}}$ in the two figures. Clearly, we find qualitatively the same behaviour of $I_{\text{inst}}(S;X)$ and $I_{\rm R}$ as in the system of $5\times 5$ spins that is investigated in the main text. Fig. 4a of the main text is constructed by retrieving the maximum value of $I_{\rm R}$ at each temperature in figure S7.

Scaling the system size

In order to investigate the behaviour of the information transmission rate as the distance $d$ between the input and output signal increases together with the system size ${\cal N}$, we have repeated the same computations in a system of $15\times 15$ spins at a distance between input and output spin of $d=6$. Fig. S8a shows the information transmission rate $I_{\rm R}$ as a function of the correlation time of the input signal $\tau_{\rm s}$ for a range of temperatures. While the sampling noise has increased due to the larger distance between the input and output spin, it is clear that there is an optimal temperature $T_{\rm opt}$ that maximizes the information transmission rate. Panel b of Fig. S8 shows this maximum information transmission rate $I_{\rm R,max}$ as a function of temperature. It is seen that there is an optimal temperature $T_{\rm opt}$ that globally maximizes the information transmission rate for this system. By fitting a quadratic function to this plot, we estimate the optimal temperature to be $T_{\rm opt}=2.38$.

Fig. S9 shows the optimal temperature as a function of system size ${\cal N}$, scaling the distance $d$ between input and output spin together with system size ${\cal N}$. The three points correspond to ($d=2,{\cal N}=5)$, ($d=3,{\cal N}=10$), and ($d=4,{\cal N}=15$). It is seen that the optimal temperature that maximizes the information transmission rate moves in the direction of the critical temperature as the system size is increased.