What we learn from the learning rate

TL;DR

This paper investigates the learning rate in bipartite Markov chains, clarifying its physical meaning and how it differs from mutual information, especially in complex systems, questioning its effectiveness as a sensor performance metric.

Contribution

The study analyzes the behavior of the learning rate in simple and complex systems, highlighting its differences from mutual information and its limitations as a sensor performance measure.

Findings

In simple systems, learning rate and mutual information are nearly equivalent.

In complex steady-state systems, they behave differently, with the learning rate reflecting update frequency.

Optimizing the learning rate can lead to sub-optimal sensor performance.

Abstract

The learning rate is an information-theoretical quantity for bipartite Markov chains describing two coupled subsystems. It is defined as the rate at which transitions in the downstream subsystem tend to increase the mutual information between the two subsystems, and is bounded by the dissipation arising from these transitions. Its physical interpretation, however, is unclear, although it has been used as a metric for the sensing performance of the downstream subsystem. In this paper, we explore the behaviour of the learning rate for a number of simple model systems, establishing when and how its behaviour is distinct from the instantaneous mutual information between subsystems. In the simplest case, the two are almost equivalent. In more complex steady-state systems, the mutual information and the learning rate behave qualitatively distinctly, with the learning rate clearly now reflecting the rate at which the downstream system must update its information in response to changes in the upstream system. It is not clear whether this quantity is the most natural measure for sensor performance, and, indeed, we provide an example in which optimising the learning rate over a region of parameter space of the downstream system yields an apparently sub-optimal sensor.

Equations

$$H[X]=-\sum _{x}p(x)\ln p(x),$$

$$H[X|Y]=-\sum _{xy}p(x,y)\ln p(x|y),$$

$$I[X;Y]=H[X]-H[X|Y]=\sum _{xy}p(x,y)\log \frac{p(x,y)}{p(x)p(y)}.$$

$$w_{yy^{\prime}}^{xx^{\prime}}=\left\{\begin{array}[]{lr}w_{y}^{xx^{\prime}}&\text{if }y=y^{\prime}\text{ and }x\neq x^{\prime},\\ w_{yy^{\prime}}^{x}&\text{if }y\neq y^{\prime}\text{ and }x=x^{\prime},\\ 0&\text{if }y\neq y^{\prime}\text{ and }x\neq x^{\prime}.\end{array}\right.$$

$${\dot{S}}_i={\mathrm{d}}_t{S}^X+{\dot{S}}_r^X-{\dot{I}}^X+{\mathrm{d}}_t{S}^Y+{\dot{S}}_r^Y-{\dot{I}}^Y\ge 0$$

$${\mathrm{d}}_t{S}^X$$

$${\mathrm{d}}_t{S}^Y$$

$${\dot{S}}_r^X$$

$${\dot{S}}_r^Y$$

$${\dot{I}}^X$$

$${\dot{I}}^Y$$

$$\frac{\mathrm{d}}{\mathrm{d}t}I[X;Y]={\dot{I}}^X+{\dot{I}}^Y.$$

$${\mathrm{d}}_t{S}^X+{\dot{S}}_r^X-{\dot{I}}^X$$

$${\mathrm{d}}_t{S}^Y+{\dot{S}}_r^Y-{\dot{I}}^Y$$

$${\dot{I}}^X+{\dot{I}}^Y=0.$$

$$\l_{Y}=\frac{\mathrm{d}}{\mathrm{d}\tau}I[X_{t};Y_{t+\tau}]\Big{|}_{\tau=0}.$$

$${\text{l}}_Y=-\sum _{xy{y}^{\prime }}{w}_{y{y}^{\prime }}^{x}p(x,y)\log \frac{p(x,y)}{p(x,y^{\prime })},$$

$$\l_{Y}=-\frac{\mathrm{d}}{\mathrm{d}\tau}I[X_{t+\tau};Y_{t}]\Big{|}_{\tau=0}.$$

$${\text{l}}_Y=$$

$$-\left((w_{10}^1{p}_{11}-w_{01}^1{p}_{10})\ln \frac{p_{11}}{p_{10}}\right).$$

$${\text{l}}_Y=\varphi \ln \frac{p_{00}{p}_{11}}{p_{01}{p}_{10}}.$$

$$\varphi$$

$$=p(Y=1|X=1)p(X=1)k_{\downarrow }$$

$$-p(Y=1|X=0)p(X=0)k_{\uparrow }.$$

$$\phi=\frac{k_{\uparrow}k_{\downarrow}}{k_{\uparrow}+k_{\downarrow}}\Big{(}p(Y=1|X=1)-p(Y=1|X=0)\Big{)}.$$

$$p_{10}$$

$$p_{00}$$

$$\frac{w_{01}^0}{w_{10}^0}=\frac{w_{10}^1}{w_{01}^1}=\frac{w_0^{01}}{w_0^{10}}=\frac{w_1^{10}}{w_1^{01}}={\mathrm{e}}^{-\beta \Delta G}.$$

$$\text{P}+\text{X}$$

$$\text{ATP}+\text{X}$$

$$\text{GTP}+\text{X}$$

$$\l_{Y}=-\dot{I}^{X}=-\sum_{xx^{\prime}y}w^{xx^{\prime}}p(x,y)\log\frac{p(y|x^{\prime})}{p(y|x)}=\sum_{xx^{\prime}y}w^{xx^{\prime}}p(x,y)\big{(}\log p(y|x)-\log p(y|x^{\prime})\big{)}.$$

$${\text{l}}_Y=\sum _{x{x}^{\prime }y}\left(p(x,y)w^{x{x}^{\prime }}-p(x^{\prime },y)w^{x^{\prime }x}\right)\log p(y|x).$$

$${\text{l}}_Y(N)=\frac{1}{2N}\sum _{xy}\left(p_N(y|x)w^{x{x}^{+}}-p_N(y|x^{+})w^{x^{+}x}+p_N(y|x)w^{x{x}^{-}}-p_N(y|x^{-})w^{x^{-}x}\right)\log p_N(y|x)$$

$$w^{x{x}^{+}}=w^{x^{-}x}=N{k}_{\uparrow },$$

Full text

I Introduction

The mathematical theory of communication was founded by Claude Shannon in 1948 Shannon_A_1948 . He was concerned with how to transfer a message from one point to another, and described the input signal through a random variable $X$ and the output by a random variable $Y.$ He introduced the Shannon entropy $H[X],$ which quantifies the a priori uncertainty in $X,$

[TABLE]

where $x$ labels the possible outcomes of the (discrete) random variable $X.$ The logarithm can be taken with respect to any base; we choose to use natural logarithms, in which case entropy is measured in nats. If a meaningful signal is passed between $X$ and $Y,$ then knowledge of $Y$ should reduce the uncertainty in $X.$ The average uncertainty in $X$ that remains given knowledge of $Y$ is given by the conditional entropy

[TABLE]

where $y$ labels the possible outcomes of $Y.$ It is necessarily true that $H[X|Y]\leq H[X]$; it is not possible to become more uncertain about $X$ by knowing $Y$ Elements_of_Information_Theory . The difference between the two quantities is the mutual information between $X$ and $Y$ Elements_of_Information_Theory ,

[TABLE]

The mutual information $I[X;Y]$ is symmetric with respect to switching $X$ and $Y.$

The terminology “entropy” is suggestive of a link with thermodynamics. In fact, the entropy $H[X]$ is precisely the generalised non-equilibrium entropy of a thermodynamic system described by $X$ crooks_entropy_1999 ; Jarzynski2011 ; Seifert2012 . Information is therefore indicative of a low entropy state, and recent work has focussed on the thermodynamic consequences of generating information Sagawa2009 ; Sagawa2012 ; still_thermodynamics_2012 ; horowitz_thermodynamics_2014 ; hartich_stochastic_2014 ; parrondo_thermodynamics_2015 ; Bo_Thermodynamic_2015 ; ouldridge_thermodynamics_2017 ; Ouldridge_polymer_2016 , and the possibility of exploiting information to do useful work Bauer2012 ; Horowitz2013 ; Chapman2015 ; McGrath_A_2017 ; Yamamoto_Linear_2016 . This body of work provides an explanation of Maxwell’s infamous demon Bennett2003 that does not rely on “erasure” of memories Landauer1961 ; Bennett2003 . The demon, by measuring its environment, appears to be able to extract work from equilibrium fluctuations. However, the demon can only extract work from its measurements if they generate non-equilibrium mutual information, and generating this mutual information in the first place requires consumption of thermodynamic resources parrondo_thermodynamics_2015 ; ouldridge_thermodynamics_2017 .

A key area in the interplay between information and thermodynamics is in biochemical sensing. Given that cells have a limited supply of resources, it is reasonable to ask whether these are put to optimally efficient use in sensing their environment — and if not, to explain why. Knowing the fundamental thermodynamic limits on sensor operation is also relevant to the engineering of synthetic sensors. Drawing on the connections between information and thermodynamics, several groups have considered the intrinsic costs associated with sensing and adaption (sensors that adapt revert to zero after lengthy exposure to a constant input) Tu2008 ; Lan2012 ; Sartori_Thermodynamic_2014 ; Mehta_Energetic_2012 ; govern2014 ; Govern_PRL_2014 ; tenWolde_Fundamental_2016 ; Mancini_Trade-Offs_2016 ; ito_maxwells_2015 ; ouldridge_thermodynamics_2017 ; Bo_Thermodynamic_2015 ; Das_A_2016 . For example, Govern and ten Wolde showed that molecular readout molecules of cell-surface receptors must consume free energy to form long-lived memories of receptor states Govern_PRL_2014 ; govern2014 , a process equivalent to thermodynamic measurement of the kind performed by Maxwell’s demon ouldridge_thermodynamics_2017 . Similarly, Bo, Del Giudice and Celani showed that the information stored by the molecular readouts about the entire trajectory of the receptors is bounded by the dissipation of the system minus the dissipation of the receptor transitions alone Bo_Thermodynamic_2015 .

A quantity, $\l_{Y},$ called the learning rate has been proposed as a metric for the performance a sensor when the signal/sensor system is modelled as a bipartite Markov chain barato_efficiency_2014 ; hartich_sensory_2016 . The learning rate is bounded by the entropy production of the sensor so this appears to put an energetic bound on sensing quality. However, it is not clear that it is the most natural measure of sensing.

In this paper, we will consider the behaviour of the learning rate for three simple steady-state systems, identifying and explaining differences between the physical content of $\l_{Y}$ and $I[X;Y].$ Typically, differences in behaviour arise because the learning rate quantifies the rate at which transitions in $Y$ must act to restore information between $X$ and $Y,$ which is not necessarily closely related to the steady-state level of information or correlation. As a result, we are able to identify a fourth system in which optimizing over a parameter of the sensor at fixed input signal dynamics gives markedly different results when using the learning rate and the mutual information as metrics. In this case, optimising the mutual information provides the more intuitively reasonable optimal sensor.

II Set up: Sensing and thermodynamics in bipartite Markov chains

Bipartite Markov chains horowitz_thermodynamics_2014 ; hartich_stochastic_2014 are a central tool in modelling Maxwell’s demon-like behaviour and cellular sensing circuits. A bipartite Markov chain has states that are labelled with two variables $x$ and $y,$ which are generally taken to correspond to the states of two subsystems $X$ and $Y.$ Transitions only change one of the variables so the transition rate from $(x,y)$ to $(x^{\prime},y^{\prime})$ is

[TABLE]

These rates have dimensions of inverse time. Throughout this paper we will use arbitrary units. The systems we will consider have a discrete state space and continuous time.

The second law of thermodynamics constrains the entropy production of the system to be positive. We can use the bipartite structure to separate the entropy production, $\dot{S}_{i}$, into six parts horowitz_thermodynamics_2014

[TABLE]

where the rates of increase of entropy of the $X$ and $Y$ subsystems are

[TABLE]

the rates of increase of the entropy of the environment due to transitions in $X$ and $Y$ are

[TABLE]

and

[TABLE]

Note that in Equation 7, if a single change of state of $X,Y$ can be associated with multiple distinct reaction pathways, these separate pathways must be explicitly considered as separate contributions to the sum Yamamoto_Linear_2016 . The flows $\dot{I}^{X}$ and $\dot{I}^{Y}$ are so called because they represent changes in the mutual information between $X$ and $Y$ due to transitions in $X$ and $Y$ respectively,

[TABLE]

Importantly, the second law holds not just for Equation 5, but also for each subsytem individually allahverdyan_thermodynamic_2009 ; horowitz_thermodynamics_2014 ; barato_efficiency_2014

[TABLE]

This form of the second law has been extended by Horowitz Horowitz_Multipartite_2015 to systems with multiple subsystems that have independent noise, and has been used by Goldt and Seifert Gold_Stochastic_2017 to find the thermodynamic bound on the information learned by a feedforward neural network with Markovian dynamics.

We can identify $\sigma^{X}=\mathrm{d}_{t}S^{X}+\dot{S}_{r}^{X}$ and $\sigma^{Y}=\mathrm{d}_{t}S^{Y}+\dot{S}_{r}^{Y}$ as the entropy productions of $X$ and $Y$, ignoring the correlations between the two subsystems. $\dot{I}^{Y}<0$ then allows for $\sigma^{Y}<0$ — correlations between subsystems allow for apparent violation of the second law if the subsystems are erroneously considered in isolation. If ${\sigma}^{Y}<0$ without direct energy flow from $X$ to $Y,$ $X$ can be interpreted as a demon acting on subsystem $Y.$ In terms of sensing, $Y$ is generally interpreted as a sensory network for the signal $X.$

In steady state the mutual information is constant and so

[TABLE]

In this case, transitions in one subsystem must reduce the mutual information as much as the transitions in the other subsystem increase it.

The quantities $\dot{I}^{Y}$ and $\dot{I}^{X}$ have been introduced in Equation 5 to separate the second law into two parts. But since $\dot{I}^{Y}$ represents the rate at which transitions in $Y$ tend to increase information between $X$ and $Y,$ it has been named the learning rate $\l_{Y}=\dot{I}^{Y}$ and proposed as a metric for the performance of $Y$ as a sensor of $X$ barato_efficiency_2014 ; hartich_sensory_2016 . Identifying $\l_{Y}$ as a measure of sensory performance also sets a thermodynamic bound on sensor function, since from Equation 10, $\l_{Y}\leq\sigma^{Y},$ the entropy generation due to transitions in $Y.$ It is straightforward to show that one can equivalently write the learning rate as hartich_sensory_2016

[TABLE]

Intuitively, $\l_{Y}$ is the degree to which future values of $Y$ are more predictive of the current value of $X$ the current value of $Y$ is, due to $Y$ continuing to learn about $X.$

Although the use of the learning rate to quantify sensor performance seems reasonable from its definition, its physical interpretation has not been extensively explored. Other authors, such as Das et al. Das_A_2016 , have preferred the instantaneous mutual information, $I[X;Y],$ over $\l_{Y}$ as a metric for performance when deriving thermodynamic constraints on sensor function. Is $\l_{Y}$ essentially reporting the degree of interdependence between $X$ and $Y,$ similar to quantities like $I[X;Y]$ or the covariance $\mathrm{Cov}(X,Y)$? Or does it highlight distinct features of the network? Is $\l_{Y}$ informative when trying to build optimal sensors for a given input signal? A further interpretational challenge occurs in steady state (a common setting for sensors). In this case barato_efficiency_2014 ; hartich_sensory_2016 ; Das_A_2016

[TABLE]

and

[TABLE]

Equation 14 suggests that learning rate also reflects “nostalgia”, a quantity defined for discrete-time systems by Still et al. still_thermodynamics_2012 . In steady state, $\l_{Y}$ not only reflects how much more the sensor $Y$ learns about the current value of $X$ as time progresses incrementally (Equation 12), but also the degree to which $Y$ has nostalgic information about the history of $X$ that is irrelevant to future signals (Equation 14). From this “nostalgia” perspective, it is unclear why large values of $\l_{Y}$ would correspond to effective sensing.

III Results and Discussion

III.1 The learning rate reports on correlations in the simplest bipartite sensing system

We first consider a canonical steady-state sensing system involving a two-state signal $X$ and a two-state sensor $Y,$ in which transition propensities for $X,$ $w^{xx\prime}_{y},$ are $Y$-independent barato_information_2013 ; barato_efficiency_2014 . This $2\times 2$ system is the simplest possible sensing setup. In biophysical terms, this system (illustrated in Figure 1) would correspond to a single receptor that can bind to ligand molecules, which is exposed to two discrete concentrations of ligands in its external environment barato_efficiency_2014 . The states $x=0,1$ correspond to low and high concentrations, and $y=0,1$ to unbound and bound receptors, respectively.

Let us, for convenience, assume that the high concentration is twice that of the low concentration. The rate at which the environment transitions from the low state to the high is $k_{\uparrow}$ and the reverse transition occurs at a rate $k_{\downarrow},$ independent of the receptor state. We only consider a single receptor molecule, which can be in two states: bound and unbound. The rate for a ligand molecule to bind to the receptor is proportional to the ligand concentration (i.e. mass-action kinetics and a well-mixed system are assumed). The concentrations are absorbed into the transition rates $k_{+}$ and $2k_{+}$ for ligand-binding in low and high concentration conditions, respectively. The rate of unbinding, $k_{-},$ is independent of the ligand concentration.

In Figure 2, we plot the instantaneous mutual information $I[X;Y],$ the learning rate $\l_{Y}$ and the covariance $\rm{Cov}(X,Y)$ as a function of $k_{+}$ and $k_{-},$ at fixed $k_{\uparrow}$ and $k_{\downarrow}$ (in this case, $k_{\uparrow}=k_{\downarrow}$). Although the functions are not identical, the dependence on $k_{+}$ and $k_{-}$ is very similar for all three. In particular, if $I[X;Y]$ is greater at $(k_{+},k_{-})$ than at $(k^{\prime}_{+},k^{\prime}_{-})$ then it is usually true that $\l_{Y}$ is greater at $(k_{+},k_{-})$ than at $(k^{\prime}_{+},k^{\prime}_{-}).$ The metrics are therefore almost equivalent for the purpose of optimising this sensor design. Deviations from this equivalence are manifest as non-parallel gradients of $I[X;Y]$ and $\l_{Y}$ with respect to $k_{+}$ and $k_{-}$; we plot the angle between gradients as a function of $k_{+}$ and $k_{-}$ in Figure 2 (d), showing that it is small.

It is possible to understand this similarity from the underlying definition of the learning rate. Using the notation $p(x,y)=p_{xy},$ then Equation 13 can be expanded as

[TABLE]

This simple system has one loop of states. In the steady state, any non-zero clockwise flux $\phi=w^{0}_{10}p_{01}-w^{0}_{01}p_{00}$ between the $(0,1)$ and $(0,0)$ states must be balanced by an equal clockwise flux for all other pairs of states around the loop, as illustrated in Figure 1. In particular, $\phi=w^{1}_{01}p_{10}-w^{1}_{10}p_{11}$ so

[TABLE]

The logarithm in Equation 16 is known as the ‘information affinity’ horowitz_thermodynamics_2014 ; Yamamoto_Linear_2016 , and it reflects the informational driving force exerted by one subsystem on the other. Clearly, it is related to the correlation between $X$ and $Y.$ If $X$ and $Y$ are more likely to be in the same state than different states then the affinity is positive. If they are more likely to be in different states then the affinity is negative.

The flux $\phi$ is the conjugate current to the information affinity horowitz_thermodynamics_2014 ; Yamamoto_Linear_2016 . For this system the flux can be written

[TABLE]

In this simple case we have assumed that $X$ transitions are independent of $Y,$ and thus $p(x)$ is determined solely by $k_{\uparrow}$ and $k_{\downarrow}$: $p(X=1)=k_{\uparrow}/(k_{\uparrow}+k_{\downarrow})$ and $p(X=0)=k_{\downarrow}/(k_{\uparrow}+k_{\downarrow}).$ Thus

[TABLE]

The quotient in Equation 18 is independent of $k_{+}$ and $k_{-}.$ It reflects the overall timescale of the process, which is manifest in the learning rate since $\l_{Y}$ is a dimensional quantity, and the uncertainty or entropy in $X.$ The second part is clearly related, again, to the correlation between $X$ and $Y$: it is large and positive when they are correlated, and large and negative when anticorrelated, just like the information affinity.

Thus, for a $2\times 2$ bipartite system, in which the value of $Y$ does not influence the transitions in $X,$ the learning rate $\l_{Y}$ essentially reports on the correlation between $X$ and $Y,$ and is thus closely related to the mutual information $I[X;Y]$ or covariance between $X$ and $Y.$ $\l_{Y}$ also incorporates a prefactor that reflects the overall timescale of the process. In Sections III.2 and III.3, we shall violate the assumptions that lead to this conclusion and explore the consequences for the relationship between learning rate and mutual information.

III.2 Feedback from the downstream to the upstream system

The simplest way to violate the assumptions of Section III.1 is to allow the signal transition rates $w^{xx^{\prime}}_{y}$ to depend on $Y,$ whilst retaining the $2\times 2$ structure of the overall system. To provide an intuition for such a setting, it is useful to consider the molecular system illustrated in Figure 3, containing two bound molecules, X and Y. Both molecules have two states; for concreteness, we imagine that X can be phosphorylated (bound to an inorganic phosphate moiety) or unphosphorylated, corresponding to states of a random variable $X=1$ and $X=0,$ respectively. Molecule Y, by contrast, can adopt two allosteric configurations, and an energetic coupling between X and Y favours states $X,Y=1,1$ and $0,0$ relative to $1,0$ and $0,1.$

If these molecules are in contact only with a bath of phosphate, they will eventually reach a thermodynamic equilibrium described by a Gibbs distribution. Assuming for simplicity that the free energies of the 11 and 00 states are equal, and $\Delta G$ below the free energies of 10 and 01,

[TABLE]

where $Z=2+2\mathrm{e}^{-\beta\Delta G}$ is the partition function, and $\beta$ is the inverse temperature $1/k_{\mathrm{B}}T.$ At equilibrium, the system must obey detailed balance:

[TABLE]

The simplest choice consistent with this requirement is $w^{0}_{01}=w^{1}_{10}=w^{01}_{0}=w^{10}_{1}=k\exp(-\beta\Delta G)$; $w^{0}_{10}=w^{1}_{01}=w^{10}_{0}=w^{01}_{1}=k.$

The transitions in $X$ are thus strongly influenced by the state of $Y$ — a natural and necessary feature in an equilibrium system involving coupling between $X$ and $Y$ Govern_PRL_2014 . One can still evaluate Equation 13 in this setting, obtaining a nominal $\l_{Y}=0$ for all parameter choices. Indeed, $\l_{Y}$ is always zero in thermodynamic equilibrium; the contributions arising from each transition in Equation 13 must cancel with those arising from the reverse transitions. Similarly, Equation 16 still holds, but the flux $\phi$ is necessarily zero due to detailed balance, even though $I[X;Y]$ and $\mathrm{Cov}(X,Y)$ can be large ($I[X;Y]=0.56$ for $\beta\Delta G=3.5$).

This equilibrium system and the driven system considered in Section III.1 are at opposite ends of a spectrum. In the equilibrium case, the influence of $X$ on $Y$ and $Y$ on $X$ are symmetric, in the sense that both experience the same biasing due to the interaction. Thus, there is no real sense in which Y is a sensor for X; it causes changes in X as much as it reacts to them. For the driven system in Section III.1, the influence is totally asymmetric; $X$ influences $Y$ but $Y$ does not influence $X$ at all, which is possible because the external process $X$ is strongly driven. In this setting Y is a purely reactionary sensor.

One might argue that the “learning rate” is no longer a meaningful term for the quantity defined by the second line of Equation 8 in the setting where $Y$ influences $X$. However, $\l_{Y}$ still quantifies the degree to which $Y$ systematically reacts to (or learns about) $X$. To see this in more detail, we now take the unusual step of interpolating between the two limits of equilibrium and strong driving. We do this by adding a finite-strength non-equilibrium driving term to the equilibrium system, to explore the response of the learning rate.

This interpolation takes the form of additional transitions of rate $q$ between X and X${}^{*},$ independent of the state of molecule Y, alongside the original transitions of rate $k$ and $k\exp(-\beta\Delta G)$. This system is illustrated in Figure 4 (a). We can vary $q/k$ to interpolate between equilibrium ($q/k\rightarrow 0$) and strongly-driven ($q/k\rightarrow\infty)$ systems; in the second case, $X$ is independent of $Y.$

A possible way to implement this scheme in a molecular setting is to drive the system by allowing phosphorylation of X to occur by two additional pathways as well as exchange of phosphate with solution. These additional pathways proceed via exchange of phosphate with nucleotides supplied by large buffers or chemostats, as shown in Figure 4 (b). With phosphate P and nucleotides ATP, ADP, GTP and GDP in solution, molecule X can change phosphorylation state by the following reactions

[TABLE]

Here, X∗ represents the molecule in the phosphorylated state. The concentrations of ATP, ADP, GTP and GDP contribute to the overall free energy changes of the second and third reactions; setting their concentrations to be incommensurate with the free energy change for $\text{P}+\text{X}\rightleftharpoons\text{X}^{*}$ amounts to driving the system so that it cannot reach equilibrium. For example, if [ADP] and [GTP] are both kept low, X will frequently be converted into X∗ through interactions with ATP, but will be converted from X∗ to X by interactions with GDP. The relative rate of these two reactions is not constrained by the relative rate of the two reactions involved in direct exchange of phosphate with solution, $\text{P}+\text{X}\rightleftharpoons\text{X}^{*},$ since the microscopic processes are distinct. Via this scheme it is therefore possible, at least in principle, to modify the stochastic process as shown in Figure 4 (a). In particular, we could imagine that ATP/ADP coupling is only possible in the $Y=0$ state, and GTP/GDP coupling is only possible when $Y=1$. The concentrations could then be set so that the free energy changes of ATP/ADP and GTP/GDP interconversion exactly cancel with the $\pm\Delta G$ associated with the change of state of $X,Y$ in these two cases, allowing phosphorylation/dephosphorylation by these pathways to have equal transition rates.

We perform the interpolation between equilibrium and driven systems in Figure 5 (a), where we have plotted the mutual information and learning rate against $q/k$ for $\beta\Delta G=3.5$ at steady state. In this system the mutual information and learning rate behave quite differently. At $q/k=0,$ $I[X;Y]$ is large and its size is set by the energy gap; as $\beta\Delta G\rightarrow\infty,$ $I[X;Y]\rightarrow\ln 2,$ corresponding to perfect correlation. As $q/k$ increases, $X$ transitions start to become increasingly decoupled from the state of $Y,$ and also much faster. Correlations are thus destroyed as shown in Figure 5 (b). The $X$ state changes randomly and too fast for $Y$ to track it and consequently, $I[X;Y]\rightarrow 0.$

By contrast, the learning rate is zero for $q/k=0,$ has a peak at $q/k\sim 1$ and decays to zero as $q/k\rightarrow\infty.$ To understand this behaviour, it is helpful to consider Equation 16. For a $2\times 2$ state system the learning rate is equal to the flux $\phi$ around the states multiplied by the information affinity. As shown in Figure 5 (c), the information affinity behaves very similarly to $I[X;Y].$ This is because, as we have discussed in Section III.1, the information affinity measures the correlation between $X$ and $Y$ and increasing $q/k$ decreases the correlation. Therefore, the ratio $\frac{p_{11}p_{00}}{p_{01}p_{10}}$ becomes closer to 1 so the information affinity decreases.

In contrast to the information affinity, the flux $\phi$ increases with increased driving strength $q/k.$ The reason for this is that as $q/k\rightarrow 0$, the system tends towards a detailed balanced equilibrium state, whereas as $q/k$ increases the system is increasingly dominated by the non-equilibrium drive, which causes a current. The combination of the opposing behaviours of the information affinity and the flux is the peaked shape of $\l_{Y}.$ It may seem surprising that increasing $q/k$ causes opposing behaviour of the information affinity and its conjugate current $\phi$. However, in general affinities depend on relative (not absolute) rates, whereas fluxes are determined by absolute rates — similar behaviour therefore arises in general systems. In any case, now that the assumption of no feedback from $Y$ to $X$ is violated, it is no longer true that $\phi$ and hence $\l_{Y}$ reflect the strength of correlation, as in Section III.1.

What intuition, then, do we gain from the learning rate? The presence of a flux $\phi>0$ is indicative of the fact that $Y$ is responding to changes in $X$; systematically, $X$ tends to transition from 0 to 1 before $Y,$ which then tends to follow. The dependence of $\l_{Y}$ on $\phi$ thus reflects the fact that the learning rate measures the degree to which transitions in $Y$ respond to transitions in $X$ to maintain correlations, rather than the strength of correlations themselves. This is apparent from the definitions in Equations 12 and 14. In a system with feedback from $Y$ to $X,$ these two metrics report on quite distinct properties. In the equilibrium limit, correlations are strong but transitions in $Y$ are just as likely to precede transitions in $X$ as to follow them, and hence the learning rate is zero.

III.3 A more complex signal process

In Section III.2, we relaxed the assumption that the downstream system $Y$ did not influence the upstream system $X.$ As a result, the tight connection between learning rate and correlation between $X$ and $Y,$ observed in Section III.1, was broken. We now explore the possibility of introducing more complex upstream signals $X$ than assumed in Section III.1, whilst restoring the assumption that $X$ is not influenced by $Y.$ We still consider systems in the stationary state.

As a physical model to provide intuition, we again consider a model of concentration sensing by a single receptor. In this case, we consider an oscillating concentration signal Becker2015 , as might be relevant to a cell experiencing roughly periodic, but noisy, fluctuations in its environment. Such a situation might be relevant to cells in animal intestines, for example. To construct an oscillating concentration, we consider a Markov process, $X$, with $2N$ discrete states, as shown in Figure 6 (a). States of $X$ with $x=1...N$ have a concentration $C=xC_{0}/N,$ and states $x=N+1...2N$ have a concentration $C=(2N+1-x)C_{0}/N.$ Thus there are $N$ distinct concentrations, each present at two values of $X.$ We consider transitions between only neighbouring $X$ states, with $w^{x\,x+1}=Nk_{\uparrow}$ and $w^{x\,x-1}=Nk_{\downarrow}.$

This partitioning of $X$ into two “rows” allows the construction of a Markov process $X$ that leads to noisy oscillations in the concentration $C,$ between $\tfrac{C_{0}}{N}$ and $C_{0}.$ Taking $Nk_{\uparrow}>Nk_{\downarrow},$ the system will tend to move to higher values of $C$ when $1\leq x\leq N,$ and towards lower values when $N+1\leq x\leq 2N$ (clockwise in Figure 6 (a)). Scaling rate constants with $N$ means that the overall flux around the cycle is $N$-independent. The typical behaviour of $X(t)$ and $C(t)$ is shown in Figure 6 (c). A similar Markov chain biased by a constant chemical potential to produce noisy oscillations was used by Barato and Seifert Barato_Cost_2016 to model a clock.

The receptor state defines the sub-process $Y;$ states and transitions are shown in Figure 6 (b). The receptor properties are identical to that considered for the simpler system in Section III.1. There is a constant rate of unbinding, $w^{x}_{10}=k_{-},$ and the binding rate $w^{x}_{01}=C(x)k_{+}$ is proportional to the ligand concentration.

From the perspective of the learning rate, it is illuminating to vary $N$ (the number of discrete concentration states) whilst fixing all other parameters. As is evident from Figure 7, $I[X;Y]$ increases with $N$ up to a finite plateau, whereas the learning rate decreases to zero. The behaviour of $I[X,Y]$ is intuitive; as more states are added, it becomes easier to reliably distinguish values of $X$ that correspond to high and low values of $C.$ In particular, for low $N,$ $C$ changes in large jumps due to $X$ transitions, meaning that $Y$ is an inaccurate reporter for $X$ immediately after the transition. For larger $N,$ the change in $C$ is smoother. Eventually, however, this effect saturates, and $I[X;Y]$ reaches a finite plateau.

By contrast, the learning rate eventually falls with large $N,$ tending to zero in the limit $N\rightarrow\infty$ (we present a general proof in Appendix A). To understand this difference in behaviour, note that our choice of $w^{x\,x+1}=Nk_{\uparrow}$ and $w^{x\,x-1}=Nk_{\downarrow}$ ensures that the overall average rate at which the concentration undergoes oscillations is fixed, regardless of $N.$ The number of forward and backward steps of the process $X$ in a time $\Delta t$ are independently Poisson-distributed with means $Nk_{\uparrow}\Delta t$ and $Nk_{\downarrow}\Delta t,$ respectively. Thus, the mean net number of forward steps in $\Delta t$ is $\Delta tN(k_{\uparrow}-k_{\downarrow}),$ proportional to $N,$ compensating for the fact there are more states covering the same concentration window. The standard deviation in the net number of forward steps in $\Delta t$ is $\sqrt{\Delta tN(k_{\uparrow}+k_{\downarrow})},$ however. Consequently, as $N$ is increased, the relative uncertainty in the net number of forward steps during $\Delta t$ falls as $1/\sqrt{N}.$ Larger $N$ therefore implies a smaller relative error in predicting the future value of $X,$ given knowledge of its current value. Increasing the number of states $N$ is thus effectively a method to interpolate between highly stochastic and quasi-deterministic behaviour of $X$. Note that it is the convergence on deterministic behaviour of the signal that in general takes $\l_{Y}\rightarrow 0$; not $N\rightarrow\infty.$

As $N\rightarrow\infty,$ therefore, the rate at which the current value of $Y$ becomes ineffective in predicting future values of $X$ tends towards zero. In other words, the “nostalgia” of $Y$ (Equation 14 and Reference still_thermodynamics_2012 ), and hence $\l_{Y},$ tends towards zero. The learning rate quantifies the rate at which transitions in $Y$ must respond to transitions in $X$ to maintain a steady-state information; this rate is zero for large $N$ despite $I[X;Y]$ increasing monotonically with $N.$ Thus, using a more complex signal than in the simple $2\times 2$ network in Section III.1 also serves to highlight the distinctions between the physical meaning of $\l_{Y}$ and $I[X;Y].$

III.4 Optimisation of sensors

Due to its provenance as an informational quantity, $\l_{Y}$ has been proposed as a metric for sensor performance in steady-state sensing circuits barato_efficiency_2014 ; hartich_sensory_2016 . In Sections III.2 and III.3, however, we have emphasised that $\l_{Y}$ reflects the rate at which transitions in the downstream subsystem $Y$ respond to changes in $X$ to maintain a steady-state information $I[X;Y].$ There is no reason a priori to suppose that maximising such a quantity is inherently optimal for a sensor — indeed, one might imagine that having to compensate for transitions in $X$ at a lower rate would be preferable in reliable sensing. Further, our results in Sections III.2 and III.3 show that the response of $\l_{Y}$ to parameter variation can be unrepresentative of the degree to which $Y$ relates to signal $X.$

Sections III.2 and III.3, however, cannot reasonably be described as sensor optimisation. To optimise a sensor, it is natural to consider a setting in which the dynamics of signal $X$ are fixed, and optimisation is performed over the parameters of the downstream network $Y.$ In Sections III.2 and III.3, however, we considered the response of $\l_{Y}$ to variations in the dynamics of $X.$ Moreover, in Section III.2, we even considered a “downstream” network $Y$ which influences the “upstream” network $X.$ This is hardly a well-defined sensor. In the simple system of Section III.1, for which we did consider variation over the parameters of the $Y$ sub-process, and $X$ was not influenced by $Y,$ optimising $\l_{Y}$ and $I[X;Y]$ over sensor parameters was essentially equivalent. It is therefore worth considering whether signal/sensor architectures do exist in which optimising over the sensor parameters for $\l_{Y}$ and $I[X;Y]$ give markedly different results.

To demonstrate this possibility, we consider a system identical to the oscillating concentration sensor considered in Section III.3, but for which the transition rates of the $X$ sub-process are $x$-dependent (Figure 8). The sensor $Y,$ with transition rates $w^{x}_{10}=k_{-}$ and $w^{x}_{01}=C(x)k_{+},$ is unchanged.

We consider optimising over the relative rates of the $X$ and $Y$ subsystems at fixed number of $X$ states $N,$ fixed transition rates in the $X$ subsystem, and fixed $k_{+}/k_{-}$ within $Y.$ In Figure 9, we show that the instantaneous mutual information increases monotonically with the rates of the downstream subsystem (represented by $k_{-}$), from $I[X;Y]=0$ up to a plateau. By contrast, the learning rate increases from $\l_{Y}=0$ to a peak at finite $k_{-},$ before decaying to a plateau. Maximising the instantaneous mutual information and maximising the learning rate give qualitatively different “optimal sensors”.

We have considered optimising over this subspace with $k_{+}/k_{-}$ fixed rather than the whole $(k_{+},k_{-})$ space for ease of illustration. For a metric to be useful for sensor optimisation it must be able to meaningfully differentiate sensing quality between any two sets of parameters, and provide a useful optimisation within any subspace of the parameter space. We note that similar behaviour is observed for other values of $k_{+}/k_{-}$.

The “optimal sensor” indicated by the instantaneous mutual information is intuitively reasonable. When the rates of the sensor are as large as possible, $Y$ can keep up with $X$; otherwise it simply lags behind, reporting earlier values of $X.$ Whilst it is possible that a slower-responding sensor could provide lower $I[X,Y],$ but increased predictive power at some specific future time $\tau,$ $I[X_{t+\tau};Y_{t}]$ Becker2015 , we find no evidence of this for our case (see Figure 9). Therefore, the instantaneous mutual information provides a metric for sensor performance that is intuitively reasonable, if not necessarily unique. By contrast, we can find no intuitive justification of the peak of the learning rate in terms of optimal sensing. Indeed, the peak in the learning rate is reminiscent of the peak in the dissipation rate $\sigma_{Y}$ that arises in such systems Becker2013 . This peak is indicative of a subsystem $Y$ that lags behind $X$ sufficiently that the response to driving is highly irreversible, but is not so slow that it barely responds at all Becker2013 . Similarly, it is plausible that the peak in $\l_{Y}$ reflects a delay in $Y$ relative to $X$ that its large enough to ensure that transitions have a strong tendency to increase $I[X;Y],$ but not so large that $Y$ barely responds to changes in $X$ at all. Regardless, since we are unable to explain the location of this peak through any optimal tracking perspective, it is difficult to see how $\l_{Y}$ is informative of quality of sensing.

IV Conclusions

We have explored the physical interpretation of the learning rate $\l_{Y},$ a recently proposed statistical metric defined on bipartite stochastic systems barato_efficiency_2014 ; hartich_sensory_2016 ; zhang_critical_2016 . $\l_{Y}$ quantifies the rate at which transitions in a downstream system $Y$ act to increase the mutual information between $Y$ and an upstream system $X,$ or the rate at which future values of $Y$ become more predictive of the current values of $X$ than the current value of $Y$. In the steady state, a biophysically relevant scenario, $\l_{Y}$ is also equivalent to the rate at which transitions in $X$ reduce the mutual information, or the “nostalgia” rate at which information about the current value of $X$ becomes irrelevant.

We have shown that, in the simplest steady-state bipartite system in which $X$ is not influenced by $Y,$ $\l_{Y}$ essentially reports on the correlation between subsystems. In other settings $\l_{Y}$ can show quantitatively and qualitatively different behaviour from measures of interdependence such as the mutual information and covariance. Moreover, we have demonstrated that this difference can be explained by the fact that $\l_{Y}$ quantifies the rate at which transitions in $Y$ act to maintain a steady state $I[X;Y],$ rather than the magnitude of $I[X;Y].$ In general, there is no reason why these two properties should behave similarly.

Fundamentally, since $l_{Y}$ represents the rate at which transitions in $Y$ act to increase information between $X$ and $Y$, the “learning rate” is a reasonable description of this quantity. However, the rate of learning and quality of sensing are not directly related, and we do not find that $\l_{Y}$ is a good general metric for sensor performance, as has been proposed barato_efficiency_2014 ; hartich_sensory_2016 . We have shown that, in at least one signal/sensor context, maximising the learning rate over a subset of sensor parameters gives a result that is hard to interpret as an optimal sensor. Without first identifying how the learning rate is reporting on sensor performance in a given context, it is difficult to see how $\l_{Y}$ can be reliably used as a metric for sensing in general.

One can imagine two sensors, $Y$ and $Y^{\prime},$ of the same signal $X,$ which produce very different joint probability distributions $p(x,y)$ and $p(x,y^{\prime})$ but the same instantaneous mutual information $I(X,Y)=I(X,Y^{\prime})$. It is perfectly possible, for example, that $Y$ predicts the future of $X$ less well than $Y^{\prime}$, despite the instantaneous mutual information being equal, if the fluctuations in $X$ coupled to by $Y$ are less informative about system dynamics than those reported by $Y^{\prime}$. In this case, to maintain equal information in the steady state, $Y$ will need to refresh its information more rapidly, and hence we would have $l_{Y}>l_{Y^{\prime}}$. However, it would seem unreasonable to classify $Y$ as a better sensor than $Y^{\prime}$; indeed, if anything, $Y^{\prime}$ seems more effective since it possesses the same information about the present state of $X$ and more information about its future evolution. High $l_{Y}$ may therefore be indicative of a sensor that is in some sense inefficient: constantly needing to update information that rapidly becomes worthless.

If $l_{Y}$ is not a direct measure of sensor performance, the fact that $l_{Y}$ is bounded by the entropy production due to $Y$-transitions, $\sigma_{Y}>l_{Y}$, does not imply that entropy production is limiting for sensing. However, we do not argue that entropy production is not limiting for sensing — only that such a justification via the learning rate would be flawed. As argued by Govern and ten Wolde Govern_PRL_2014 , in any equilibrium system, there is no real distinction between “signal” and “readout” — both must necessarily influence each other, and therefore the construction of a true sensor is impossible. In Reference Das_A_2016 a bound relating entropy generation to instantaneous mutual information is derived in the case where the sensor does not influence the signal.

Our work has considered only a small set of possible Markovian systems, and has not touched on issues such as performance of readout networks that are designed to integrate a signal over time govern2014 ; ouldridge_thermodynamics_2017 or perform more complex operations such as differentiation Becker2015 . It is possible that the learning rate reflects an aspect of the performance in these circuits. Further, we have not investigated the behaviour of the learning rate in systems that are yet to reach steady state. In such cases, $\l_{Y}$ is no longer equal to either the nostalgia rate, or the rate at which transitions in $X$ tend to reduce $I[X;Y].$ Moreover, it is potentially non-zero even for passive (non-driven) systems. The meaning of the learning rate in such cases warrants further consideration.

Finally, the learning rate can provide insight into the functioning of systems even if it is not a metric for performance of sensing or similar functionality. For example, its properties were recently applied to derive results that do not depend on its interpretation in a simple model of a perceptron Gold_Stochastic_2017 . $\l_{Y}$ can also reasonably be interpreted as an indicator of the direction and magnitude of information flow in a steady-state bipartite network, as evidenced by its role in the refined second law (Equation 10). In the context of a model of an autonomous Maxwell’s demon it is the flow of information between the ‘demon’ subsystem and the ‘engine’ subsystem horowitz_thermodynamics_2014 .

One interpretation of the learning rate in steady state might be as a quantification of the effort put in by the downstream system to maintain a certain interdependence of $X$ and $Y,$ regardless of whether this interdependence is optimal in some sense. For example, if the magnitude of all rates is increased but their ratios kept constant, then the learning rate is increased although there is no change in the correlation between the sensor and signal. Indeed, this sense of $l_{Y}$ as a measure of how hard the downstream system has to work is apparently reinforced by the relation $l_{Y}<\sigma_{Y}$, with $\sigma_{Y}$ being the entropy generation due to transitions in the Y-subsystem. It should be noted, however, that although entropy is generated by $Y$ transitions, this is not necessarily indicative of a consumption of resources held by the $Y$ subsystem barato_efficiency_2014 . In the biomolecular examples given here, for example, the $Y$ subsystems are totally passive, consuming no chemical fuel. The thermodynamic work is done by the driving of the external process $X$ barato_efficiency_2014 , which is particularly evident in the example of Section III.2, where the nucleotides fuelling the driving are explicitly considered. From the perspective of a sensor or a cell, entropy generation arising from environmental changes, rather than due to internal fuel consumption, is not obviously a limiting factor.

V Acknowledgements

We acknowledge Susanne Still for a discussion of the topics and Pieter Rein ten Wolde, Andre Barato, Udo Seifert, David Hartich and Sebastian Goldt for commenting on the manuscript.

T. E. O. acknowledges support from a Royal Society University Research Fellowship and R. A. B. acknowledges support from an Imperial College London AMMP studentship.

Appendix A Proof that $\l_{Y}$ tends to zero as $N\rightarrow\infty$ for the system in Section III.3

Using Equations 8 and 11, in the steady state

[TABLE]

Relabelling dummy variables,

[TABLE]

In this system, $p(x)=\tfrac{1}{2N}$ for all $x$ by symmetry. There are only transitions between adjacent $X$ states and so

[TABLE]

where $x^{+}$ is the next $X$ state in the cycle and $x^{-}$ is the previous state, and we have introduced an explicit dependence of probabilities on $N.$ In this system

[TABLE]

and therefore

[TABLE]

Defining $z=\tfrac{x}{2N}$ we have

[TABLE]

We now assume that for large $N,$ $p_{N}(y|z)$ tends smoothly towards $p_{\infty}(y|z)$, where $p_{\infty}(y|z)$ is a smooth function of $z$ with $p_{\infty}(y|0)=p_{\infty}(y|1)$ due to the boundary conditions that connect $z=1$ ($x=2N$) to $z=0$ ($x=0$). For sufficiently large $N,$

[TABLE]

Thus

[TABLE]

and the sum can be approximated by an integral

[TABLE]

So for large $N,$

[TABLE]

Performing the integral within the sum gives

[TABLE]

where the second equality follows from $p_{\infty}(y|0)=p_{\infty}(y|1)$. Therefore, $\l_{Y}(N)\sim\mathcal{O}\left(\frac{1}{N}\right)$ and thus tends to zero in the limit of $N\rightarrow\infty.$