Information Content and Maximum Entropy of Compartmental Systems in Equilibrium

TL;DR

This paper introduces an information-theoretic framework to analyze equilibrium compartmental systems, enabling model selection and revealing structural properties.

Contribution

The novel contribution is extending the maximum entropy principle to compartmental dynamics and deriving path entropy and entropy rates for such systems.

Findings

Closed-form expressions for path entropy and entropy rates are derived in equilibrium.

The framework reveals hidden structural properties of complex systems like the global carbon cycle.

It provides a systematic approach to address equifinality in model selection.

Abstract

Mass-balanced compartmental systems defy classical deterministic entropy measures since both metric and topological entropy vanish in dissipative dynamics. By interpreting open compartmental systems as absorbing continuous-time Markov chains that describe the random journey of a single representative particle, we allow established information-theoretic principles to be applied to this particular type of deterministic dynamical system. In particular, path entropy quantifies the uncertainty of complete trajectories, while entropy rates measure the average uncertainty of instantaneous transitions. Using Shannon’s information entropy, we derive closed-form expressions for these quantities in equilibrium and extend the maximum entropy principle (MaxEnt) to the problem of model selection in compartmental dynamics. This information-theoretic framework not only provides a systematic way to address equifinality but also reveals hidden structural properties of complex systems such as the global carbon cycle.

Figures

Full text

1. Introduction

For many modeling applications it is of interest to quantify the complexity of the system of differential equations used to represent natural phenomena [1,2]. In principle, we are interested in selecting models that are parsimonious, i.e., models with the least degree of complexity for explaining certain patterns in nature [3]. The concept of entropy has been commonly used to characterize complexity or information content. Classical entropy measures for dynamical systems characterize the rate of increase in dynamical complexity as the system evolves over time [4]. These metrics have been used extensively to characterize chaotic behavior in complex nonlinear systems [5], but they give trivial results for a large range of models used in the natural sciences.

In a large variety of scientific fields, models are based on the principle of mass conservation. In many cases, such models are nonnegative dynamical systems that can be described by first-order systems of ordinary differential equations (ODEs) with strong structural constraints. Such systems are called compartmental systems [6,7,8].

Compartmental systems can be evaluated using diagnostic metrics that predict system-level behavior and allow comparisons of systems of very different structures. Age and transit time of material content in compartmental systems are two diagnostic metrics that have been widely studied for systems in and out of equilibrium [9,10,11,12,13,14]. They help compare the behavior and quality of different models. Nevertheless, structurally very different models might show very similar ages and transit times and might represent a given measurement equally well. If we are in the position to select one of such models, which is the one to select? This equifinality problem can be resolved by the maximum entropy principle (MaxEnt) [15,16], a generic procedure to draw unbiased inferences from measurement or stochastic data [3,17].

In order to apply MaxEnt to compartmental systems, an appropriate notion of entropy is required to measure the system’s uncertainty or information content. Two classical examples in dynamical systems theory are the topological entropy and the Kolmogorov–Sinai/metric entropy. However, open compartmental systems are dissipative, and trajectories with slightly disturbed initial conditions do not diverge. Hence, by Pesin’s theorem [18], both metric and topological entropy vanish and cannot serve as a measure of uncertainty. Alternatively, we can interpret compartmental systems as weighted directed graphs. Dehmer and Mowshowitz [19] provide a comprehensive overview of the history of graph entropy measures. Unfortunately, most such entropy measures are based on the number of vertices, vertex degree, edges, or degree sequence [20,21]. Thus, they concentrate only on the structural information of the graph. There are also graph theoretical measures that take edges and weights into account by using probability schemes. Their drawback is that the underlying meaning of uncertainty becomes difficult to interpret, because the assigned probabilities seem somewhat arbitrary [22].

To bridge this gap, we interpret deterministic compartmental systems from a probabilistic viewpoint which allows us to apply the whole information theoretical toolbox to this important class of deterministic systems. As a first step in this direction, we compute the Shannon information entropy [23] of the continuous-time Markov chain that describes the random path of a single particle through the compartmental system [13] and introduce three non-vanishing entropy measures: While the path entropy describes the uncertainty of a single particle’s path through the system, the entropy rate per unit time and the entropy rate per jump describe average uncertainties over the course of a particle’s journey.

The focus on a single particle gives our entropies microscopic system properties and consequently distinguishes our approach from the theory of maximum caliber (MaxCal) [24,25], where path entropy is interpreted as a macroscopic system property of bulk material. Furthermore, our approach differs from the thermodynamic approach to entropy, which has been developed by other authors studying energy transfers and reversibility in thermodynamic systems [8,26,27,28]. While the probabilistic interpretation of thermodynamic entropy is related to the uncertainty of the location of a typical particle at a specific point in time, the newly introduced path entropy considers all locations of a typical particle at all times while it is part of the system. A first application of our information theoretical entropy concept to compartmental systems allows us to reveal hidden inherent properties of complex systems such as the carbon cycle, from the microbial to the global scale, e.g., allowing us to partly explain why there is a large diversity of soil carbon models, while there is more consensus on how to model carbon uptake by photosynthesis.

This article is organized as follows. First, we provide the fundamentals from information theory and dynamical systems theory that are necessary to introduce path entropy as the uncertainty of a single particle traveling through the system. Then, we mathematically derive the path entropy and introduce the entropy rates per unit time and per jump as uncertainty measures of the behavior of one typical particle. The focus on a single particle gives our entropies microscopic system properties and provides insights, where the macroscopic approach via topological and metric entropy fails. Then, we prove that the new entropy rates of a finite particle path are indeed proper entropy rates of an associated stationary stochastic process, which guarantees that, in the process of model selection, there exists a unique optimal solution as long as the parameter set is convex. To assist in the interpretation of the newly introduced quantities from a system-wide point of view, we establish the link to the macroscopic system scale before we introduce the link between MaxEnt and structural model identification for compartmental systems. Afterwards, we present the introduced theory by means of simple generic examples from the field of carbon-cycle modeling exploring the effect of different parameterizations on the three entropy metrics before we apply MaxEnt to a model identification problem. Then, we discuss the results and draw final conclusions.

2. Mathematical Background: Information Entropy and Compartmental Systems as Markov Chains

First, we introduce some basic notations and well-known properties of Shannon information entropy of random variables and stochastic processes. Then, we present compartmental systems as a means to model material-cycle systems that obey the law of mass balance. We then consider such systems from a single-particle point of view and define the path of a single particle through the system along with its visited compartments, sojourn times, occupation times, and transit time.

2.1. Short Summary of Shannon Information Entropy

We introduce a few basic concepts of information entropy. Within the framework of this article, discrete entropies are usually associated with a particle’s jump into another compartment and differential entropies to a particle’s sojourn time within a specific compartment. Entropy rates are defined as average uncertainties of the particle’s path while it is in the system. See (Cover and Thomas [29] Sects. 2 and 8) for a more detailed introduction to Shannon’s information entropy and differential entropy. Entropy rates for discrete- and continuous-time stochastic processes are introduced in (Cover and Thomas [29] Sect. 4) and Bad Dumitrescu [30].

Let Y be a real-valued discrete (continuous) random variable and call p its probability mass function (probability density function). Then,

$$[eqn]$$

is called the Shannon information entropy (differential entropy) of Y. Most of the time, we just say entropy, and the precise meaning can be derived from the context. The entropy’s unit depends on the logarithmic base. For base 2, the unit is bits, and for the natural logarithm with base e, the unit is nats. Throughout this manuscript, we use the latter if not stated otherwise.

The entropy $[eqn]$ of a random variable Y has two intertwined interpretations. On the one hand, it is a measure of uncertainty, i.e., a measure of how difficult it is to predict the outcome of a realization of Y. On the other hand, $[eqn]$ is also a measure of the information content of Y, i.e., a measure of how much information we gain once we learn about the outcome of a realization of Y. It is important to note that, even though their definitions and information theoretical interpretations are quite similar, Shannon and differential entropy have one main difference. Shannon entropy is always nonnegative, whereas differential entropy can have negative values. While Shannon entropy is an absolute measure of information and makes sense in its own right, differential entropy is not an absolute information measure, is not scale-invariant, and makes sense only in comparison with the differential entropy of another random variable.

Panel (a) of Figure 1 depicts the Shannon entropy with logarithmic base 2 of a Bernoulli random variable Y, with $[eqn]$ representing a coin toss with probability of heads equal to p. The closer p is to $[eqn]$ , the more difficult it is to predict the outcome. For an unbiased coin with $[eqn]$ , we have no information about the outcome whatsoever, and the Shannon entropy

$$[eqn]$$

is maximized. Panel (b) of Figure 1 shows the differential entropy of an exponentially distributed random variable $[eqn]$ with rate parameter $[eqn]$ , probability density function $[eqn]$ for $[eqn]$ , and $[eqn]$ . We can imagine it to represent the duration of stay of a particle in a well-mixed compartment in an equilibrium compartmental system, where $[eqn]$ is the total outflow rate from the compartment. The higher the outflow rate, the more likely an early exit of the particle, and the easier it is to predict its moment of exit. Hence, the differential entropy

$$[eqn]$$

decreases with increasing $[eqn]$ .

The joint entropy of two random variables $[eqn]$ and $[eqn]$ can be described as

$$[eqn]$$

where the conditional entropy  $[eqn]$ describes the uncertainty of $[eqn]$ under the condition that $[eqn]$ is known. The uncertainty of a stochastic process can be measured by its entropy rate, which describes the time density of the average information in the process. For a discrete-time stochastic process $[eqn]$ , it is defined as ([29] Sect. 4.2)

$$[eqn]$$

when the limit exists.

For instance, let $[eqn]$ be a Poisson process with intensity rate $[eqn]$ describing the moments of occurrences of certain events. The interarrival times $[eqn]$ of Z (the times between events) are $[eqn]$ -distributed and mutually independent. Hence, $[eqn]$ . If we rescale $[eqn]$ by the mean interarrival time $[eqn]$ , we obtain the entropy rate of Z being ([31] Sect. 3.3)

$$[eqn]$$

This entropy rate increases with $[eqn]$ , reaches its maximum at 1, and then it decreases (Figure 1c). The maximum always occurs at $[eqn]$ independent of the unit of $[eqn]$ , because it is based on the differential entropy of the exponential distribution, which is not scale-invariant. Consequently, it is not an absolute measure of information content but only useful in comparison to the entropy rates of other stochastic processes.

2.2. Compartmental Systems in Equilibrium

The mass-balanced flow of material into a system, within the system, and out of the system that consists of several compartments can be modeled by so-called compartmental systems [6,32]. Compartments are always well-mixed and usually also called pools or boxes. An autonomous compartmental system can be described by the d-dimensional linear ODE system

$$[eqn]$$

with some nonnegative initial condition $[eqn]$ . The nonnegative vector $[eqn]$ describes the amount of material in the different compartments at time t, the nonnegative vector $[eqn]$ is the vector of external inputs to the compartments, and the compartmental matrix $[eqn]$ describes the flux rates between the compartments and out of the system. The nonnegative off-diagonal value $[eqn]$ is the flux rate from compartment j to compartment i, the absolute value of the negative diagonal value $[eqn]$ is the total rate of fluxes out of compartment j, and the nonnegative column sum $[eqn]$ is the rate of the flux from compartment j out of the system. By requiring $[eqn]$ to be invertible, we ensure that the system is open, i.e., all material that enters the system will eventually also leave it. Throughout this manuscript, we consider the open compartmental system (7) to have reached its unique steady-state or equilibrium compartment vector $[eqn]$ . This implies $[eqn]$ , where $[eqn]$ given by $[eqn]$ is the external outflux vector from the system, and $[eqn]$ denotes the sum of absolute values of a vector ( $[eqn]$ -norm). An open compartmental system in equilibrium given by Equation (7) is fully characterized by $[eqn]$ and $[eqn]$ , and we denote it by $[eqn]$ .

2.3. The One-Particle Perspective

While Equation (7) describes the movement of bulk material through the system, compartmental systems in equilibrium can also be described probabilistically by considering the random path of a single particle through the system [13]. If $[eqn]$ denotes the compartment in which the single particle is at time t, and $[eqn]$ if the particle has already left the system, then $[eqn]$ is an absorbing continuous-time Markov chain [33] on $[eqn]$ . Its initial distribution is given by $[eqn]$ , where $[eqn]$ , and hence, $[eqn]$ is the probability of the single particle to enter the system through compartment j. The superscript T denotes the transpose of the respective vector or matrix. The transition-rate matrix of X is given by

$$[eqn]$$

and thus,

$$[eqn]$$

is the probability of the particle to be in compartment i at time t if $[eqn]$ or that the particle has left the system if $[eqn]$ . Here, $[eqn]$ denotes the matrix exponential. Furthermore,

$$[eqn]$$

is the probability that X is in state i at time t, given it was in state j at a previous time s. Since the Markov chain X and the compartmental system in equilibrium given by Equation (7) are equivalent, we can write

$$[eqn]$$

2.4. The Path of a Single Particle

A particle’s path through the system from the moment of entering until the moment of exit can be described as a sequence of (compartment, sojourn-time)-pairs

$$[eqn]$$

where X is the absorbing Markov chain associated with the particle’s journey. The sequence $[eqn]$ represents the successively visited compartments with the associated sojourn times $[eqn]$ , and the random variable

$$[eqn]$$

denotes the first hitting time of the absorbing state $[eqn]$ by the embedded jump chain $[eqn]$ of X [33]. With $[eqn]$ , the one-step transition probabilities of Y are given by, for $[eqn]$ ,

$$[eqn]$$

Let $[eqn]$ be the restriction of $[eqn]$ to S. We can also write $[eqn]$ , where $[eqn]$ is the diagonal matrix with the diagonal entries of $[eqn]$ , and $[eqn]$ denotes the identity matrix of appropriate dimension. Then, $[eqn]$ is the fundamental matrix of Y. The entry $[eqn]$ denotes the expected number of visits to compartment i given that the particle entered the system through compartment j. Consequently, the expected number of visits to compartment $[eqn]$ is given by

$$[eqn]$$

and the total expected number of jumps is given by

$$[eqn]$$

where we take into account also the last jump out of the system.

The last jump, $[eqn]$ , leads the particle out of the system such that, at the moment of this last jump, X takes on the value $[eqn]$ . This last jump happens at the absorption time of the Markov chain X, which is defined as

$$[eqn]$$

The absorption time is phase-type distributed [34], $[eqn]$ , with probability density function

$$[eqn]$$

It can be shown ([13] Sect. 3.2) that the mean or expected value of $[eqn]$ equals the turnover time [12] of system (7) in equilibrium and is given by total stocks over total fluxes, i.e.,

$$[eqn]$$

Furthermore, by construction, $[eqn]$ . If we denote by $[eqn]$ the indicator function of the logical expression A given by

$$[eqn]$$

then $[eqn]$ is the total time that the particle spends in compartment j. This time is called occupation time of j, and its mean is given by ([13] Sect. 3.3)

$$[eqn]$$

which induces $[eqn]$ .

3. Entropy Measures, MaxEnt, and Structural Model Identification

Based on these basic structures of the path of a single particle traveling through the system, we compute three different types of entropy, for which we provide a summary of the desirable relations among them below:

• (1)As a particle travels through a system of interconnected compartments, it jumps a certain number of times to the next compartment until it finally jumps out of the system. Between two jumps, the particle resides in some compartment. The path entropy measures the entire uncertainty about the particle’s travel through the system, including both the sequence of visited compartments and the respective times spent there.
• (2)The entire travel of the particle takes a certain time. In each unit time interval before the particle leaves the system, uncertainties exist as to whether the particle jumps, where it jumps, and even how often it jumps. The mean of these uncertainties over the mean length of the travel interval is measured by the entropy rate per unit time.
• (3)Each jump comes with uncertainties about which compartment will be next and how long will the particle stay there. The entropy rate per jump measures the average of these uncertainties with respect to the mean number of jumps until the particle’s exit from the system.

Once these entropy metrics are established, we introduce MaxEnt and show how to apply it to the problem of structural model identification.

3.1. Path Entropy, Entropy Rate per Unit Time, and Entropy Rate per Jump

The path $[eqn]$ given by Equation (12) can be interpreted in three different ways. Each of these ways leads to a different interpretation of the path’s entropy. First, we can look at $[eqn]$ as the result of bookkeeping of the absorbing continuous-time Markov chain X, where for all times t we note down the pair $[eqn]$ of the current compartment and the current time. Second, we can consider the path as a discrete-time process. In each time step n, we choose randomly a new compartment $[eqn]$ and an associated sojourn time $[eqn]$ of the particle in this compartment. Third, we can look at $[eqn]$ as a single random variable with values in the space of all possible paths. Based on the latter interpretation we now derive the path entropy.

We are interested in the uncertainty/information content of the path $[eqn]$ of a single particle. Along the lines of Albert [35], we construct a space ℘ that contains all possible paths that can be taken by a particle that runs through the system until it leaves. Let $[eqn]$ denote the space of paths that visit n compartments/states before ending up in the environmental compartment/absorbing state $[eqn]$ . By $[eqn]$ , we denote the space of all eventually absorbed paths. Note that, since $[eqn]$ is invertible, a path through the system is finite with probability 1. Let l denote the Lebesgue measure on $[eqn]$ and c the counting measure on S. Furthermore, let $[eqn]$ be the $[eqn]$ -finite product measure on $[eqn]$ . It is defined by $[eqn]$ . Almost all sample functions of $[eqn]$ can be represented as a point $[eqn]$ ([36] Chapter VI). Consequently, we can represent X by a finite-length path $[eqn]$ for some $[eqn]$ , where $[eqn]$ .

For each set $[eqn]$ , for which $[eqn]$ is $[eqn]$ -measurable for each $[eqn]$ , we define $[eqn]$ . This measure is defined on the $[eqn]$ -field $[eqn]$ which is the smallest $[eqn]$ -field containing all sets $[eqn]$ , whose projection on $[eqn]$ is a Borel set for each $[eqn]$ . Let $[eqn]$ be a measure on all sample functions, defined for all subsets W whose intersection with ℘ is in $[eqn]$ . We define it by $[eqn]$ .

Let $[eqn]$ for some $[eqn]$ . For $[eqn]$ , denote by $[eqn]$ the total number of path’s p one-step transitions from j to i and by $[eqn]$ the total amount of time spent in j.

Theorem 1. The probability density function of $[eqn]$ with respect to σ is given by

$$[eqn]$$

Proof. Let $[eqn]$ , $[eqn]$ , and $[eqn]$ . Since

$$[eqn]$$

with $[eqn]$ , the probability density function of $[eqn]$ with respect to $[eqn]$ is given by

$$[eqn]$$

The term $[eqn]$ enters exactly $[eqn]$ times. Furthermore,

$$[eqn]$$

We make the according substitutions, and the proof is finished. □

The entropy of the absorbing continuous-time Markov chain X is equal to its entropy on the random but finite time horizon $[eqn]$ , which in turn equals the entropy of a single particle’s path $[eqn]$ through the system.

Theorem 2. The entropy of the absorbing continuous-time Markov chain X is given by

$$[eqn]$$

Proof. Let X have the finite path representation

$$[eqn]$$

for some $[eqn]$ , and denote by $[eqn]$ its probability density function. Then, by Theorem 1,

$$[eqn]$$

We compute the expectation and get

$$[eqn]$$

Obviously, $[eqn]$ is the mean occupation time of compartment $[eqn]$ by X. Furthermore, for $[eqn]$ and $[eqn]$ such that $[eqn]$ , by Equations (14) and (15),

$$[eqn]$$

Together with $[eqn]$ , we obtain

$$[eqn]$$

□

By some simple substitutions and rearrangements, we obtain two representations of $[eqn]$ that are easy to interpret. For simplicity of notation, we define

$$[eqn]$$

Proposition 1. The entropy of the absorbing continuous-time Markov chain X is also given by

$$[eqn]$$

and

$$[eqn]$$

which can be rewritten as

$$[eqn]$$ $$[eqn]$$

Proof. By virtue of Equation (33), we replace $[eqn]$ by $[eqn]$ in Equation (26) and take into account that the entropy rate of a Poisson process with intensity rate $[eqn]$ equals $[eqn]$ to prove Equation (33). To prove Equation (34), we use Equation (15) to replace $[eqn]$ in Equation (26) by $[eqn]$ and obtain

$$[eqn]$$

Here, $[eqn]$ is the entropy of an exponential random variable with rate parameter $[eqn]$ . Using definition (14) of $[eqn]$ , we replace $[eqn]$ by $[eqn]$ for $[eqn]$ and $[eqn]$ by $[eqn]$ and finish the proof. □

By identifying a compartmental system $[eqn]$ with its associated absorbing continuous-time Markov chain X and the according path $[eqn]$ of a single traveling particle, we transfer the concept of the path entropy $[eqn]$ from the probabilistic to the deterministic realm.

Definition 1. The path entropy of the compartmental system M in equilibrium given by Equation (7) with associated absorbing continuous-time Markov chain X and path $[eqn]$ is defined by

$$[eqn]$$

Consider a one-dimensional compartmental system $[eqn]$ in equilibrium with rate $[eqn]$ and positive external input given by

$$[eqn]$$

and denote its associated path by $[eqn]$ . The entropy of the initial distribution vanishes, and we obtain

$$[eqn]$$

which equals the differential entropy $[eqn]$ of the exponentially distributed mean transit time $[eqn]$ , reflecting that the only uncertainty of the particle’s path in a one-pool system is the time of the particle’s exit. The exponential distribution with rate parameter $[eqn]$ is the distribution of the interarrival time of a Poisson process wit intensity rate $[eqn]$ . Hence, we can interpret $[eqn]$ as the instantaneous Poisson entropy rate $[eqn]$ multiplied with the expected duration $[eqn]$ of the particle’s stay in the system.

For a d-dimensional system, we can interpret $[eqn]$ as the entropy of a continuous-time process in the context of Equation (33) and as the entropy of a discrete-time process in the context of Equation (34). In both interpretations, the first term $[eqn]$ represents the uncertainty of the first pool through which the particle enters the system. In the continuous-time interpretation, the uncertainty of the subsequent travel is the weighted average of the superposition of d Poisson processes describing the instantaneous uncertainty of possible jumps of the particle inside the system, $[eqn]$ , and out of the system, $[eqn]$ , where the weights are the expected occupation times of the different compartments $[eqn]$ . In the discrete-time interpretation, the subsequent travel’s uncertainty is the average of uncertainties associated to each pool, weighted by the number of visits to the respective pools. The uncertainty associated with each pool comprises the uncertainty of the length of the stay in the pool, $[eqn]$ , and the uncertainty of where to jump afterwards, $[eqn]$ . Hence, in the context of Equation (34), we can separate the path entropy into a discrete part associated with the jump uncertainty given by Equation (35) and a continuous part associated with the sojourn time uncertainty given by Equation (36).

The two interpretations of the path entropy $[eqn]$ (as a continuous-time or discrete-time process) motivate two different entropy rates as described earlier. The entropy rate per unit time is given by

$$[eqn]$$

and the entropy rate per jump by

$$[eqn]$$

While the path entropy measures the uncertainty of the entire path, entropy rates measure the average uncertainty of the instantaneous future of a particle while it is in the system: for the entropy rate per unit time, it is the uncertainty entailed by the infinitesimal future, and for the entropy rate per jump, it is the uncertainty entailed by the next jump.

For these entropy rates to be useful in the process of model selection, it is important that they guarantee the existence of a unique maximum entropy model in case of a convex parameter space, which is not obvious from their definitions. The classical entropy rate of a stochastic process as defined in (Cover and Thomas [29] Sect. 4.1) has this property, and we prove in Appendix A that $[eqn]$ , where the stationary process $[eqn]$ on the space $[eqn]$ describes the infinite journey of a typical particle. It is the sequence of visited compartments with the associated sojourn times of a single particle through the system with immediate jumps back into the system when leaving it. By Equations (41) and (42), the average time between two jumps is $[eqn]$ . If we divide the entropy rate per jump by it, we obtain the entropy rate per unit time. Hence,

$$[eqn]$$

is the average uncertainty per unit time of the stationary process Z.

3.2. From Microscopic Particle Entropy to Macroscopic System Entropy

While the microscopic entropy measures provide direct insights into the uncertainties of the path of a single traveling particle, we can also scale them up to the macroscopic system scale. The combination of Equations (19), (41), and (43) inevitably leads to the following macroscopic definition.

Definition 2. The system entropy of the compartmental system M in equilibrium given by Equation (7) with associated absorbing continuous-time Markov chain X and path $[eqn]$ is defined by

$$[eqn]$$

Consequently, the system entropy can be interpreted in three ways: (1) as the cumulated mean instantaneous uncertainty of all particles currently in the system, (2) as the cumulated uncertainty of the entire future path of all particles currently entering the system, and (3) as the cumulated mean uncertainty of all future jumps of all particles currently entering the system.

3.3. The Maximum Entropy Principle (MaxEnt)

MaxEnt arose in statistical mechanics as a variational principle to predict the equilibrium states of thermal systems and later was applied to matters of information and as a general procedure to draw inferences based on self-consistency requirements [17]. Its relationship to information theory and stochastics was established by Jaynes [15,16]. The general idea is to identify the most uninformed probability distribution to represent some given data in the sense that the maximum entropy distribution, constrained to given data, uses the information provided by the data only and nothing else. This approach ensures that no additional subjective information creeps into the distribution. For compartmental systems, data constraints could affect macroscopic quantities such as the stocks $[eqn]$ , the input vector $[eqn]$ , the output rates $[eqn]$ , or the mean transit time $[eqn]$ . The goal of this section is to transfer MaxEnt to compartmental systems in order to identify the compartmental system that best represents our state of knowledge in different situations and, at the same time, to get a better understanding of the previously introduced entropy measures. In the next two examples, we identify compartmental models with maximum entropy under some restrictions. Both examples show that maximizing entropy means also maximizing symmetry as much as the given constraints allow.

Example 1. Consider the set $[eqn]$ of equilibrium compartmental systems (7) with a predefined nonzero input vector $[eqn]$ , a predefined mean transit time $[eqn]$ , and an unknown steady-state vector $[eqn]$ comprising nonzero components. We are interested in the most unbiased compartmental system that reflects our state of information, where maximum unbiasedness is achieved by identifying $[eqn]$ with the path $[eqn]$ such that the path entropy $[eqn]$ or, equivalently, the entropy rate per unit time $[eqn]$ is maximized. We can show (see Proposition A2) that the compartmental system $[eqn]$ with

$$[eqn]$$

where $[eqn]$ , is the maximum entropy model in $[eqn]$ . In the special case $[eqn]$ for a one-dimensional compartmental system, we obtain $[eqn]$ . Since, in this case, $[eqn]$ , we see that the exponential distribution is the maximum entropy distribution in the class of all nonnegative continuous probability distributions with fixed expected value. This special case is very well-known ([29] Example 12.2.5).

Example 2. Let us consider the subclass $[eqn]$ of compartmental models from the previous example with the additional restriction of a predefined positive steady-state vector $[eqn]$ . Then, the compartmental system $[eqn]$ with path $[eqn]$ and

$$[eqn]$$

is the maximum entropy model in $[eqn]$ (see Proposition A3).

3.4. Structural Model Identification Assisted by MaxEnt

Suppose that we observe a natural system and conduct measurements from which we try to construct a linear autonomous compartmental model in equilibrium that represents the observed natural system as well as possible. The first question that arises is about the number of compartments that the model should ideally have. MaxEnt cannot be helpful here because by adding more and more compartments we can theoretically increase the entropy of the model indefinitely. Consequently, the problem of finding the right dimension of system (7) has to be solved by other means. One way to do this is to analyze an impulse response function of the system and its Laplace transform, i.e., the transfer function of the system, and identify the most dominant frequencies. The impulse response or the transfer function might be possible to obtain by tracer experiments [6,37].

Once the desired number of compartments is identified, we can focus on the structure and values of external input and output fluxes as well as internal fluxes. In (Anderson [6] Chapter 16), the structural identification problem of linear autonomous systems is described as follows. Suppose that we are interested in determining a d-dimensional system of form (7). We are interested in sending an impulse to the system at time $[eqn]$ and analyzing its further behavior. To that end, we rewrite the system as

$$[eqn]$$

Note that the roles of $[eqn]$ and $[eqn]$ are interchanged here with respect to Anderson [6]. In a typical tracer experiment, we choose an input vector $[eqn]$ and the input distribution matrix $[eqn]$ , which defines how the input vector enters the system. Then, we decide which compartments we can observe to determine the output connection matrix  $[eqn]$ . The experiment is now to inject an impulse into the system and to record the output function $[eqn]$ . Bellman and Åström [38] pointed out that the input–output relation is given by

$$[eqn]$$

where ∗ is the convolution operator. The model parameters enter the input–output relation only in the matrix-valued impulse response function

$$[eqn]$$

or in the transfer function

$$[eqn]$$

which is the Laplace transform matrix of $[eqn]$ . Consequently, all identifiable parameters of $[eqn]$ , $[eqn]$ , and $[eqn]$ must be identified through $[eqn]$ or $[eqn]$ . Difficulties arise because the entries of the matrices $[eqn]$ and $[eqn]$ are usually nonlinear expressions of the elements of $[eqn]$ , $[eqn]$ , and $[eqn]$ . We call system (47) identifiable if this nonlinear system of equations has a unique solution $[eqn]$ for given $[eqn]$ or $[eqn]$ . Otherwise, the system is called non-identifiable. Usually, the matrices $[eqn]$ and $[eqn]$ are already known from the experiment setup. What remains is to identify the compartmental matrix $[eqn]$ , and this can be achieved by MaxEnt.

4. Application to Particular Systems

First, we apply the presented theory to some equilibrium compartmental models with very simple structure in order to grasp the new entropy concepts. Then, we compute entropy quantities for two carbon-cycle models in dependence on environmental and biochemical parameters. Finally, we apply MaxEnt to solve an equifinality problem in model selection as an example of how to tackle this problem arising from, for instance, tracer experiments.

4.1. Simple Examples

From Table 1, we can see that, depending on the connections between compartments, smaller systems can have greater path entropy and entropy rates than larger systems, even though systems with more compartments can theoretically reach higher entropy. Furthermore, we see from the depicted examples that the system with the highest path entropy does not have the highest entropy rate per unit time or per jump. Adding connections to a system, one would expect higher path entropy, but the path entropy might actually decrease, because the new connections potentially provide a faster way out of the system.

4.2. A Linear Autonomous Global Carbon-Cycle Model

We consider the global carbon-cycle model introduced by Emanuel et al. [39] (Figure 2).

The model comprises five compartments: non-woody tree parts $[eqn]$ , woody tree parts $[eqn]$ , ground vegetation $[eqn]$ , detritus/decomposers $[eqn]$ , and active soil carbon $[eqn]$ . We introduce an environmental rate modifier $[eqn]$ , which controls the speed at which carbon is cycled in all compartments. If $[eqn]$ , carbon is cycled faster in all compartments, simulating the effect of the global surface temperature increase [40]. For a given $[eqn]$ , the equilibrium model $[eqn]$ is given by

$$[eqn]$$

and

$$[eqn]$$

where the numbers are chosen as in Thompson and Randerson [41]. The input vector is expressed in units of petagrams of carbon per year ( $[eqn]$ ) and the fractional transfer coefficients in units of per year ( $[eqn]$ ). Because $[eqn]$ is a lower triangular matrix, the model contains no feedbacks. For every value of $[eqn]$ , the system has a different steady state (Figure 3a).The higher the value of $[eqn]$ , the faster the system, which makes the mean transit time (Figure 3b) decrease, and because of shorter paths, the path entropy (Figure 3d) also decreases. Since $[eqn]$ has no impact on the structure of the model, the mean number of jumps (Figure 3c) remains unaffected. This can also be seen from the solid line marked by squares in (Figure 3d). It represents the part of the path entropy related to jump-associated uncertainties (Equation (35)). The solid line marked by circles represents the part of the path entropy related to sojourn-associated uncertainties (Equation (36)), which as a weighted average of one-pool entropies decreases similarly to the entropy of an exponential distribution with an increasing rate parameter $[eqn]$ (Figure 1b). The two parts together constitute the path entropy as represented by the unmarked solid line.

The entropy rate per unit time (Figure 3e) increases until $[eqn]$ and decreases afterwards, because with increasing system speed the decreasing uncertainty associated with sojourn times increasingly dominates the uncertainty associated with jumps. While the uncertainty associated with jumps averaged over the path length increases, because the total jump uncertainty is constant (see solid line marked with squares in Figure 3d), and the mean path length decreases (Figure 3b), the sojourn-associated uncertainty decreases with the increasing system speed for $[eqn]$ , similar to the entropy rate of a Poisson process with intensity rate $[eqn]$ (see Figure 1c). The entropy rate per jump (Figure 3f) decreases with increasing $[eqn]$ , because the path entropy of the system decreases.

Dashed lines in Figure 3d–f show the respective entropy values for a one-pool system $[eqn]$ with the same mean transit time, i.e., $[eqn]$ . The solid and dashed lines intersect at $[eqn]$ in Figure 3d,e. Before this break-even point, the path of this multiple-pool model is harder to predict than the path (i.e., the exit time of the particle) of a one-pool model with the same mean transit time. After this point of breaking even, the path of the model with five compartments is easier to predict than only the transit time in a one-pool model. The reason is that, as the system becomes faster, the differential entropy of the sojourn times in slow pools decreases so fast that, at some point, the sojourn times in slow pools visited by few particles becomes rather unimportant. The one-pool model’s path becomes relatively harder to predict, because it puts too much weight on a small amount of slowly cycling particles.

Note that there is no point in comparing jump-associated uncertainties (square-marked lines) with one-pool entropies (dashed lines), because the former are discrete entropies and the latter differential entropies. Comparison of a differential entropy with another quantity only becomes reasonable if a second differential entropy is involved, as is true for the path entropy or the entropy rates $[eqn]$ and $[eqn]$ (unmarked solid lines). Hence, square- and circle-marked lines assist in understanding the composition of the entropies of the multi-pool system, and only the composition of the two can then be compared to the one-pool entropy rate.

4.3. A Nonlinear Autonomous Soil Organic Matter Decomposition Model

Consider the nonlinear two-compartment model $[eqn]$ , described by Wang et al. [42], which is used to represent the dynamics of microbes and carbon substrates in soils (Figure 4). Its ODE system is given by

$$[eqn]$$

where $[eqn]$ . We denote by $[eqn]$ and $[eqn]$ substrate organic carbon and soil microbial biomass carbon ( $[eqn]$ ), respectively, by $[eqn]$ the carbon use efficiency or fraction of assimilated carbon that is converted into microbial biomass (unit-less), by $[eqn]$ the turnover rate of microbial biomass per year ( $[eqn]$ ), by $[eqn]$ the carbon influx into the soil ( $[eqn]$ ), and by $[eqn]$ and $[eqn]$ the maximum rate of soil carbon assimilation per unit microbial biomass per year ( $[eqn]$ ) and the half-saturation constant for soil carbon assimilation by microbial biomass ( $[eqn]$ ), respectively.

We consider the model in equilibrium, i.e., $[eqn]$ , with

$$[eqn]$$

The equilibrium stocks depend on the carbon use efficiency $[eqn]$ and so does the compartmental matrix $[eqn]$ , because

$$[eqn]$$

From Wang et al. [42], we take the parameter values $[eqn]$ , $[eqn]$ , and $[eqn]$ = 53,954.83 $[eqn]$ . Since the description of $[eqn]$ is missing in the original publication, we let it be equal to $[eqn]$ to approximately meet the given steady-state contents $[eqn]$ = 12,650.00 $[eqn]$ and $[eqn]$ for the original value $[eqn]$ . Otherwise, we leave the carbon use efficiency $[eqn]$ as a free parameter.

In contrast to the system from the first example, this system exhibits a feedback. This feedback results from dead soil microbial biomass being considered as new soil organic matter. The feedback can also be recognized by noting that $[eqn]$ is not triangular. For every value of $[eqn]$ , the system has a different steady state (Figure 5a). The higher the value of $[eqn]$ , the lower the equilibrium substrate organic carbon and the higher the microbial biomass carbon. Caused by the model’s nonlinearity expressed in Equation (54), the system speed increases, and the mean transit time goes down (Figure 5b) with increasing $[eqn]$ . At the same time, higher carbon use efficiency increases the probability of each carbon atom to be reused more often; hence, the mean number of jumps increases (Figure 5c), making the entropy rate per jump decrease (Figure 5f). Even though the average paths become shorter, with increasing carbon use efficiency, the path entropy increases as well for most values of $[eqn]$ . This has two reasons. First, the mean uncertainty of where to jump from $[eqn]$ increases; this uncertainty decreases then for $[eqn]$ (solid line marked by squares in Figure 5f). Second, the rate $[eqn]$ of leaving the substrate pool is increasing and smaller than 1. The corresponding Poisson process reaches its maximum entropy rate at an intensity rate equal to 1 (Figure 1c), which corresponds to $[eqn]$ . This is also reflected in the entropy rate per unit time (Figure 5e). The maximum does not exactly occur at $[eqn]$ , because the time that the particle stays in the different pools also depends on $[eqn]$ . For $[eqn]$ approaching 1, both the path entropy and the entropy rate rapidly decline as the sojourn-associated uncertainties (solid lines with circle markers) decline sharply because of a nonlinear increase in the rate $[eqn]$ of soil organic carbon turnover.

Considering a one-pool system $[eqn]$ with the same mean transit time, we recognize only small sensitivity of the entropies on $[eqn]$ , because the contrary effects on path length and jump- and sojourn-associated uncertainties mostly balance out (dashed lines in Figure 5d–f).

4.4. Model Identification via Maxent

The following example is inspired by (Anderson [6] Example 16 C). It shows how MaxEnt can help make a decision about which model to use if not all parameters can be uniquely determined from the transfer function $[eqn]$ . We are interested in determining the entries of the compartmental matrix $[eqn]$ belonging to the two-dimensional equilibrium compartmental system

$$[eqn]$$

We immediately notice that $[eqn]$ and $[eqn]$ . Further, we decide to measure the contents of compartment 1 such that $[eqn]$ . We recall $[eqn]$ and obtain $[eqn]$ and $[eqn]$ . The real-valued transfer function is then given by

$$[eqn]$$

where

$$[eqn]$$

We assume that $[eqn]$ is known from measurements, i.e., $[eqn]$ , $[eqn]$ , and $[eqn]$ are known impulse response parameters. We have the four unknown parameters $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ , or equivalently, $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ , but only three equations to determine them. Consequently, the system is non-identifiable and there remains a class $[eqn]$ of models which all satisfy Equation (57). Which model out of $[eqn]$ are we going to select now?

Here, MaxEnt comes into play. We intend to select the model that best represents the information given by our measurement data. We have to find $[eqn]$ such that

$$[eqn]$$

Maximizing the entropy rate per unit time here leads to a feasible optimization problem, whereas maximization of the path entropy by slowing down the model and indefinitely increasing its mean transit time and its path entropy would lead to an unbounded optimization problem. The parameter space associated with $[eqn]$ is given by

$$[eqn]$$

which is not guaranteed to be convex in general. Consequently, by fundamental principles from mathematical optimization theory, the existence and uniqueness of $[eqn]$ are not guaranteed, and we must apply optimization methods tailored to the specific case at hand.

Let us turn to a numerical example in which we suppose to be given $[eqn]$ , $[eqn]$ , and $[eqn]$ . Since convexity of the parameter space is not guaranteed, local optimality does not guarantee global optimality. Hence, we run local optmizations from starting points on a grid with mesh side $[eqn]$ over the subspace $[eqn]$ of the parameter space and select our global maximum candidate as the local maximum with the highest entropy rate per unit time. Even though we cannot rigorously prove that our global maximum candidate $[eqn]$ , as represented by the red dot in Figure 6 with

$$[eqn]$$

and $[eqn]$ , is a global maximum, we can clearly see that it is a good candidate. Increasing the distance of the local maximum parameters (Figure 6a) and mean transit time (Figure 6b) from the global maximum candidate leads to a decrease in the entropy rate per unit time. Furthermore, local optimizations with starting points on the grid lead only to small improvements. A good choice of starting point on the grid is crucial to find a good global maximum candidate (Figure 6c). Finally, the global maximum candidate for the entropy rate per unit time does not maximize the path entropy (Figure 6d).

5. Discussion

Based on the stochastic path that a single particle takes through a deterministic compartmental system, we introduced three types of entropy based on Shannon’s information theory. The entropy of the particle’s entire path through the system is the central concept, and the entropy rates per unit time and per jump are consistently derived from it. Even though we call $[eqn]$ the path entropy and identify models by maximizing it, it is different from the concept of path entropy as treated in the context of maximum caliber (MaxCal) [24,25]. We maximize here the Shannon information entropy of a single particle’s microscopic path through a compartmental system by means of an absorbing continuous-time Markov chain, whose transition probabilities are already determined by the macroscopic equilibrium state of the system. As discussed by Pressé et al. [17], MaxCal interprets the path entropy as a macroscopic system property to be maximized in order to identify a time-dependent trajectory of the entire dynamical system, not just one single particle. We derive macroscopic system entropy by multiplying microscopic entropy quantities (e.g., path entropy, entropy rate per unit time) with the associated macroscopic system quantities (e.g., total system content, total input amount).

In the field of soil carbon cycle modeling, Ågren [43] applied the maximum entropy principle to identify the distribution of soil carbon qualities within the framework of the continuous-quality theory. Given only the nonnegative mean quality, an application of MaxEnt leads to an exponential quality distribution, because, under these circumstances, the exponential distribution is the maximum entropy distribution. The path entropy generalizes this approach to several interconnected compartments and jumps between them, while each sojourn time in a compartment is exponentially distributed.

From the simple examples in Section 4.1, we can see that models can be ordered differently in terms of uncertainty, depending on whether the interest is in the uncertainty of the entire path or in some average uncertainty rate. For applications of MaxEnt without restrictions on the transit time, it is often useful to maximize an entropy rate instead of the path entropy, because, by slowing the system down more and more, the path entropy can potentially be increased indefinitely, and a maximum path entropy model does not exist. The decision to maximize a rate can potentially also be justified by a given macroscopic restriction on the stock sizes, which would increase indefinitely with indefinitely increasing path entropy by slowing down the system.

By virtue of its very mathematical definition (Equation (1)), entropy is maximized when the system’s symmetry is maximized. This is indicated by the Bernoulli entropy (Figure 1a) and supported by Example 1. Intuitively, this result is obvious. If a system has high symmetry, a particle is equally likely to jump among different pools. The Poisson process with intensity rate 1 is the one with maximum entropy rate, which follows directly from properties of the function $[eqn]$ . Furthermore, the resulting rates $[eqn]$ of leaving the system are chosen such that the mean transit time constraint is fulfilled. In Example 2, the symmetry is broken by the additional restriction of a given steady-state vector. Consequently, $[eqn]$ .

When we compute entropy values for actual carbon-cycle models (Section 4.2 and Section 4.3), we note that environmental or eco-physiological factors might impact model entropies. For example, higher global surface temperatures are likely to induce a higher global carbon-cycle system speed ( $[eqn]$ ). This higher system speed reduces the uncertainty of the long-term future of entire paths of carbon atoms entering the terrestrial biosphere from the atmosphere. At the same time, it increases the entropy rate per unit time, i.e., the uncertainty of the short-term future of carbon atoms already in the terrestrial biosphere.

Furthermore, we see that for sufficiently fast systems, a multi-pool model has lower entropy than a one-pool model with the same system speed. The one-pool system might put too much weight on the uncertainties of a small number of slow-cycling particles, while the more detailed multi-pool model focuses more on the small uncertainties of the major amount of fast-cycling particles. The path of a detailed model that separates fast from slow paths is then even easier to predict than a one-pool model path, even though the detailed model’s path looks more complicated. However, detailed paths of slow-cycling systems are harder to predict than just the exit time in a one-pool equivalent.

The two carbon-cycle models (Section 4.2 and Section 4.3) are well-understood in equilibrium; hence, they can serve as a means to better understand properties of the newly introduced entropy metrics. Once we understand entropy properties in dependence on general system properties, we can extrapolate this understanding to far more complex systems and make qualitative statements about their predictability without going into all model details. One major insight from those two examples is that, in general, slow heterogeneous systems are much harder to predict than fast homogeneous systems. Slowness increases the uncertainty of the duration of particle’s stay in the system, and heterogeneity increases the uncertainty of a particle’s sequence of visited compartments.

These simple insights allow us to understand modeling issues in a broader sense. For instance, path entropies support the understanding of differences in the diversity of modeling approaches and predictions for carbon uptake and transfers to soils in terrestrial ecosystems. Both photosynthesis [44] and soil carbon turnover [45] are modeled by many different approaches. However, in ecosystem models, photosynthesis is almost exclusively represented [46] by the Farquhar model [47], while soil carbon dynamics are represented by a great variety of models with very different structures [48]. The latter leads to large variations in the prediction of future land carbon uptake [48,49]. A comparison of carbon simulations from eleven modeling centers showed that across models, global soil carbon varied more than twice as much as global net primary productivity [50]. Carbon dynamics in leaves are relatively fast, and the fate of this carbon is highly predictable; it is either used to fuel the metabolic activity of leaf cells or allocated to storage reserves or woody tissue [51]. In contrast, the fate of carbon entering soils is much less predictable, with a large range of potential metabolic pathways through microbial food webs or potential physico-chemical interactions with the mineral surfaces in the soil matrix that occur at longer timescales [52]. Consequently, the higher uncertainty of soil carbon cycling compared to photosynthetic carbon uptake is an inherent property of the system. Simply by the soil’s heterogeneous and slow-cycling nature, the system posseses high inherent uncertainty, which hints at a theoretical limit that cannot be overcome by any model.

The example of model identification by MaxEnt in Section 4.4 shows a major difference from the more artificial previous maximum entropy examples. The given constraints do not tell us enough about the structure of the model class $[eqn]$ to ensure that an identified local maximum is also a global maximum. Owing to the nonlinear restrictions on the parameters in Equation (57), the parameter space is probably not convex. Hence, local maxima are not guaranteed to be also globally optimal. The small system size allows us to identify a reasonable global maximum candidate model by brute force, starting local maximizations on a grid over a parameter sub-space. Practical examples might include higher-dimensional systems and thus not be feasible for brute-force approaches. More sophisticated optimization methods suitable for the particular problem at hand should then be applied. However, since the newly introduced entropy measures are proper entropies, and in the case of a convex parameter space, the existence of a unique global optimum is guaranteed.

6. Conclusions

A probabilistic approach to mass-balanced deterministic systems allows basic information theoretical principles to compute the uncertainty of a wide range of models representing complex processes in nature—a task at which classical deterministic theories fail. The information content of autonomous compartmental systems in equilibrium can be assessed by the entropy of the path of particles traveling through the system of interconnected compartments. When a particle moves through a compartmental system, it creates a path from the time of its entry until the time of its exit. This path can be described in three ways: (1) as a random variable in the path space, (2) as a continuous-time stochastic process representing the occupied compartments, (3) as a discrete sequence of pairs consisting of visited compartments and associated sojourn times. Based on these three possible descriptions, we introduced, for systems in equilibrium, (1) the entropy of the entire path, (2) the entropy rate per unit time, and (3) the entropy rate per jump. These three different entropies allow us to quantify how difficult it is to predict the path of particles entering a compartmental system, serving as a measure of system uncertainty/predictability. With these measures, it is thus possible to apply maximum entropy principles to compartmental systems in equilibrium in order to address problems of equifinality in model selection.

Although the path entropy concept developed here only applies to systems in equilibrium, it sets the foundation for future research on systems out of equilibrium. This can be achieved by building on the concept of the entropy rate per unit time as an instantaneous uncertainty and interpreting non-autonomous compartmental systems as inhomogeneous Markov chains. This would allow an extension of MaxCal, which was so far applied only to the inhomogeneous embedded jump chain, as by Ge et al. [53], to incorporate sojourn times in different compartments as well.

By introducing the concept of path entropy to compartmental systems, we made a first crucial step toward a quantification of information content in models that can be compared to other methods to obtain information content from observations. Using entropy measures based on Shannon information theory in both models and observations, we can potentially advance toward better methods for model selection applying the maximum entropy principle.