Invariant components of synergy, redundancy, and unique information among three variables

TL;DR

This paper extends the Partial Information Decomposition framework to fully characterize three-variable dependencies without relying on source-target classifications, unveiling an invariant structure that captures all interdependencies.

Contribution

It introduces a general, target/source classification-independent PID structure with seven nonnegative information atoms that fully describe three-variable dependencies.

Findings

Identifies an invariant PID structure applicable to any three-variable system.

Splits redundancy into source and non-source components based on correlations.

Provides a complete characterization of interdependencies through the invariant structure.

Abstract

In a system of three stochastic variables, the Partial Information Decomposition (PID) of Williams and Beer dissects the information that two variables (sources) carry about a third variable (target) into nonnegative information atoms that describe redundant, unique, and synergistic modes of dependencies among the variables. However, the classification of the three variables into two sources and one target limits the dependency modes that can be quantitatively resolved, and does not naturally suit all systems. Here, we extend the PID to describe trivariate modes of dependencies in full generality, without introducing additional decomposition axioms or making assumptions about the target/source nature of the variables. By comparing different PID lattices of the same system, we unveil a finer PID structure made of seven nonnegative information subatoms that are invariant to different target/source classifications and that are sufficient to construct any PID lattice. This finer structure naturally splits redundant information into two nonnegative components: the source redundancy, which arises from the pairwise correlations between the source variables, and the non-source redundancy, which does not, and relates to the synergistic information the sources carry about the target. The invariant structure is also sufficient to construct the system's entropy, hence it characterizes completely all the interdependencies in the system.

Figures

Equations

$$I(X:Y)=SI(X:\{Y;Z\})+UI(X:\{Y\backslash Z\})=SI(Y:\{X;Z\})+UI(Y:\{X\backslash Z\}),$$

$$I(X:Y|Z)=CI(X:\{Y;Z\})+UI(X:\{Y\backslash Z\})=CI(Y:\{X;Z\})+UI(Y:\{X\backslash Z\}),$$

$$RSI(X\stackrel{\text{Z}}{\leftrightarrow }Y):=$$

$$RCI(X\stackrel{\text{Z}}{\leftrightarrow }Y):=$$

$$RUI(X\stackrel{\text{Z}}{\leftrightarrow }Y):=$$

$$IRSI(X\stackrel{\text{Z}}{\leftarrow }Y):=SI(X:\{Y;Z\})-RSI(X\stackrel{\text{Z}}{\leftrightarrow }Y).$$

$$IRSI(X\stackrel{\text{Z}}{\leftarrow }Y)$$

$$=CI(X:\{Y;Z\})-CI(Y:\{X;Z\})$$

$$=UI(Y:\{X\backslash Z\})-UI(X:\{Y\backslash Z\}).$$

$$coI(X;Y;Z)=SI(i:\{j;k\})-CI(i:\{j;k\}),$$

$$\begin{array}{rl}SR(T:\{S_1;S_2\})& :=\max \{RSI(T\stackrel{S_2}{\leftrightarrow }{S}_1),RSI(T\stackrel{S_1}{\leftrightarrow }{S}_2)\}\\ & =\max \{\min \left[SI(T:\{S_1;S_2\}),SI(S_1:\{S_2;T\})\right],\\ & \min \left[SI(T:\{S_1;S_2\}),SI(S_2:\{S_1;T\})\right]\}.\end{array}$$

$$NSR(T:\{S_1;S_2\}):=SI(T:\{S_1;S_2\})-SR(T:\{S_1;S_2\}).$$

$$SR(T:\{S_1;S_2\})\le I(S_1:S_2).$$

$$H(X,Y)=I(X:Y)+H(X|Y)+H(Y|X).$$

$$H_{(1)}=H(X|Y,Z)+H(Y|X,Z)+H(Z|Y,X).$$

$$DTC\equiv H(X,Y,Z)-H_{(1)}.$$

$$DTC=I(X:Y|Z)+I(Y:Z|X)+I(X:Z|Y)+coI(X;Y;Z),$$

$$H(X,Y,Z)$$

$$+CI(X:\{Y;Z\})+UI(Y:\{Z\backslash X\})+CI(Y:\{X;Z\}),$$

$$H(X,Y,Z)-H_{(1)}$$

$$+RUI(X\stackrel{\text{Z}}{\leftrightarrow }Y)+RUI(Y\stackrel{\text{X}}{\leftrightarrow }Z)+RUI(X\stackrel{\text{Y}}{\leftrightarrow }Z)+$$

$$+2IRSI(Z\stackrel{\text{Y}}{\leftarrow }X)+3IRSI(X\stackrel{\text{Z}}{\leftarrow }Y).$$

$$SI(X:\{Y;Z\})=\min [I(X:Y),I(X:Z)].$$

$$SR(X:\{Y;Z\})=\min [I(X:Y),I(X:Z),I(Y:Z)],$$

$$NSR(X:\{Y;Z\})=\min [I(X:Y),I(X:Z)]-\min [I(X:Y),I(X:Z),I(Y:Z)].$$

$$SR(X:\{Y;Z\})=\{\begin{array}{ll}SI(X:\{Y:Z\}),& \text{if}I(Y:Z)\ge SI(X:\{Y;Z\}),\\ I(Y:Z),& \text{otherwise};\end{array}$$

$$NSR(X:\{Y;Z\})=\{\begin{array}{ll}0,& \text{if}I(Y:Z)\ge SI(X:\{Y;Z\}),\\ SI(X:\{Y:Z\})-I(Y:Z),& \text{otherwise}.\end{array}$$

$$RSI(X\stackrel{\text{Z}}{\leftrightarrow }Y)=I(X:Y),$$

Full text

1 Introduction

Shannon’s mutual information [1] provides a well established, widely applicable tool to characterize the statistical relationship between two stochastic variables. Larger values of mutual information correspond to a stronger relationship between the instantiations of the two variables in each single trial. Whenever we study a system with more than two variables, the mutual information between any two subsets of the variables still quantifies the statistical dependencies between these two subsets; however, many scientific questions in the analysis of complex systems require a finer characterization of how all variables simultaneously interact [2, 3, 4, 5, 6]. For example, two of the variables, $A$ and $B$, may carry either redundant or synergistic information about a third variable $C$ [7, 8, 9], but considering the value of the mutual information $I((A,B):C)$ alone is not enough to distinguish these qualitatively different information-carrying modes. To achieve this finer level of understanding, recent theoretical efforts have focused on decomposing the mutual information between two subsets of variables into more specific information components (see e.g. [6, 10, 11, 12]). Nonetheless, a complete framework for the information-theoretic analysis of multivariate systems is still lacking.

Here we consider the analysis of trivariate systems, which is complex enough to present the fundamental challenges of going beyond bivariate analyses, and yet simple enough to provide hints about how these challenges might be addressed. Characterizing the fine structure of the interactions among three stochastic variables can improve the understanding of many interesting problems across different disciplines [13, 14, 15, 16]. For example, in the study of neural information processing many important questions can be cast as trivariate information analyses. Determining quantitatively how two neurons encode information about an external sensory stimulus [7, 8, 9] requires describing the dependencies between the stimulus and the activity of the two neurons. Determining how the stimulus information carried by a neural response relates to the animal’s behaviour [17, 18, 19] requires the analysis of the simultaneous three-wise dependencies among the stimulus, the neural activity and the subject’s behavioral report. More generally, a thorough understanding of even the simplest information-processing systems would require the quantitative description of all different ways two inputs carry information about one output [20].

In systems where legitimate assumptions can be made about which variables act as sources of information and which variable acts as the target of information transmission, the partial information decomposition (PID) [10] provides an elegant framework to decompose the mutual information that one or two (source) variables carry about the third (target) variable into a finer lattice of redundant, unique and synergistic information atoms. However, in many systems the a priori classification of variables into sources and target is arbitrary, and limits the description of the distribution of information within the system [21]. Furthermore, even when one classification is adopted, the PID atoms do not characterize completely all the possible modes of information sharing between the sources and the target. For example, two sources can carry redundant information about the target irrespective of the strength of the correlations between them and, as a consequence, the PID redundancy atom can be larger than zero even if the sources have no mutual information  [22, 3, 23]. Hence, the value of the PID redundancy measure cannot distinguish how the correlations between two variables contribute to the information that they share about a third variable.

In this paper, we address these limitations by extending the PID framework without introducing further axioms or assumptions about the trivariate structure to analyze. We compare the atoms from the three possible PID lattices that are induced by the three possible choices for the target variable in the system. By tracking how the PID information modes change across different lattices, we move beyond the partial perspective intrinsic to a single PID lattice and unveil the finer structure common to all PID lattices. We find that all lattices can be fully described in terms of a unique minimal set of seven information-theoretic quantities, that is invariant to different classifications of the variables.

The first result of this approach is the identification of two nonnegative subatomic components of the redundant information that any pair of variables carries about the third variable. The first component, that we name source redundancy ($SR$), quantifies the part of the redundancy which arises from the correlations of the sources. The second component, that we name non-source redundancy ($NSR$), quantifies the part of the redundancy which is not related to the source correlations. Interestingly, we find that whenever the non-source redundancy is larger than zero then also the synergy is larger than zero. The second result is that the minimal set induces a unique nonnegative decomposition of the full joint entropy $H(X,Y,Z)$ of the system. This allows us to dissect completely the distribution of information of any trivariate system in a general way that is invariant with respect to the source/target classification of the variables. To illustrate the additional insights of this new approach, we finally apply our framework to paradigmatic examples, including discrete and continuous probability distributions. These applications confirm our intuitions and clarify the practical usefulness of the finer PID structure.

2 Preliminaries and state of the art

Williams and Beer proposed an influential axiomatic decomposition of the mutual information $I(X:(Y,Z))$ that two stochastic variables $Y,Z$ (the sources) carry about a third variable $X$ (the target) into the sum of four nonnegative atoms  [10]:

• •

$SI(X:\{Y;Z\})$, which is the information about the target that is shared between the two sources (the redundancy);

• •

$UI(X:\{Y\backslash Z\})$ and $UI(X:\{Z\backslash Y\})$, which are the separate pieces of information about the target that can be extracted from one of the sources, but not from the other;

• •

$CI(X:\{Y;Z\})$, which is the complementary information about the target that is only available when both of the sources are jointly observed (the synergy).

This construction is commonly known as the Partial Information Decomposition (PID). Sums of subsets of the four PID atoms provide the classical mutual information quantities between each of the sources and the target, $I(X:Y)$ and $I(X:Z)$, and the conditional mutual information quantities whereby one of the sources is the conditioned variable, $I(X:Z|Y)$ and $I(X:Y|Z)$. Such relationships are displayed with a color code in Fig. 1.

The PID decomposition of Ref. [10] is based upon a number of axioms that do not determine univocally the value of the four PID atoms. The specific measure proposed in Ref. [10] to calculate such values has been questioned as it can lead to unintuitive results [22], and thus many attempts have been devoted to finding alternative measures [24, 22, 25, 26, 27, 28] compatibly with an extended number of axioms, such as the identity axiom proposed in [22]. Other work has studied in more detail the lattice structure that underpins the PID, indicating the duality between information gain and information loss lattices [12]. Even though there is no consensus on how to build partial information decompositions in systems with more than two sources, for trivariate systems the measures of redundancy, synergy and unique information defined in Ref. [11] have not yet been questioned and have found wide acceptance (in this paper, we will in fact make use of these measures when a concrete implementation of the PID will be required).

However, even in the trivariate case there are open problems regarding the understanding of the PID atoms in relation to the interdependencies within the system. First, Harder et al. [22] pointed out that the redundant information shared between the sources about the target can intuitively arise from the following two qualitatively different modes of three-wise interdependence:

• •

the source redundancy, which is redundancy which ’must already manifest itself in the mutual information between the sources’111Note that Ref. [22] interchangeably refers to the sources as ’inputs’: we will discuss this further in Section  4 when addressing the characterization of source redundancy.;

• •

the mechanistic redundancy, which can be larger than zero even if there is no mutual information between the sources.

As pointed out by Harder and colleagues [22], a more precise conceptual and formal separation of these two kinds of redundancy still needs to be achieved, and presents fundamental challenges. The very notion that two statistically independent sources can nonetheless share information about a target was not captured by some earlier definitions of redundancy [24, 25]. Further, several studies  [23, 29] described the property that the PID measures of redundancy can be positive even when there are no correlations between the sources as undesired. On a different note, other authors [20] pointed out that the two different notions of redundancy can define qualitatively different modes of information processing in (neural) input-output networks.

Other issues were recently pointed out by James and Crutchfield [21], who indicated that the very definition of the PID lattice prevents its use as a general tool for assessing the full structure of trivariate (let alone multi-variate) statistical dependencies. In particular, Ref. [21] considered dyadic and triadic systems, which underlie quite interesting and common modes of multivariate interdependencies. They showed that, even though the PID atoms are among the very few measures that can distinguish between the two kinds of systems, a PID lattice cannot allot the full joint entropy $H(X,Y,Z)$ of either system. The decomposition of the joint entropy in terms of information components that reflect qualitatively different interactions within the system has also been subject of recent research [30, 27].

In summary, the PID framework, in its current form, does not yet provide a satisfactorily fine and complete description of the distribution of information in trivariate systems. The PID atoms do assess trivariate dependencies better than Shannon’s measures, but they cannot quantify interesting finer interdependencies within the system, such as the source redundancy that the sources share about the target. In addition, they are limited to describing the dependencies between the chosen sources and target, thus enforcing a certain perspective on the system that does not naturally suit all systems.

3 More PID diagrams unveil finer structure in the PID framework

To address the open problems described above, we begin by pointing out the feature of the PID lattice that underlies all the issues in the characterization of trivariate systems outlined in Sec. 2. As we illustrate in Fig. 1(a), while a single PID diagram involves the mutual information quantities that one or both of the sources (in the figure, $Y$ and $Z$) carry about the target $X$, it does not contain the mutual information between the sources $I(Y:Z)$ and their conditional mutual information $I(Y:Z|X)$. This precludes the characterization of source redundancy with a single PID diagram, as it prevents any comparison between the elements of the PID and $I(Y:Z)$. Moreover, it also signals that a single PID lattice cannot account for the total entropy $H(X,Y,Z)$ of the system.

These considerations suggest that the inability of the PID framework to provide a complete information-theoretic description of trivariate systems is not a consequence of the axiomatic construction underlying the PID lattice. Instead, it follows from restricting the analysis to the limited perspective on the system that is enforced by classifying the variables into sources and target when defining a PID lattice. We thus elaborate that significant progress can be achieved, without the addition of further axioms or assumptions to the PID framework, if one just considers, alongside the PID diagram in Fig. 1(a), the other two PID diagrams that are induced respectively by labeling $Y$ or $Z$ as the target in the system. When considering the resulting three PID diagrams (Fig. 2), the previously missing mutual information $I(Y:Z)$ between the original sources of the left-most diagram is now decomposed into PID atoms of the middle and the right-most diagrams in Fig. 2, and the same happens with $I(Y:Z|X)$.

In the following we take advantage of this shift in perspective to resolve the finer structure of the PID diagrams and, at the same time, to generalize its descriptive power going beyond the current limited framework, where only the information that two (source) variables carry about the third (target) variable is decomposed. More specifically, even though the PID relies on setting a partial point of view about the system, we will show that describing how the PID atoms change when we systematically rotate the choice of the PID target variable effectively overcomes the limitations intrinsic to one PID alone.

3.1 The relationship between PID diagrams with different target selections

To identify the finer structure underlying all the PID diagrams in Fig. 2, we first focus on the relationships between the PID atoms of two different diagrams, with the goal of understanding how to move from one perspective on the system to another. The key observation here is that, for each pair of variables in the system, their mutual information and their conditional mutual information given the third variable appear in two of the PID diagrams. This imposes some constraints relating the PID atoms in two different diagrams. For example, if we consider $I(X:Y)$ and $I(X:Y|Z)$, we find that:

[TABLE]

where the first and second equality in each equation result from the decomposition of $I(X:(Y,Z))$ (left-most diagram in Fig. 2) and $I(Y:(X,Z))$ (middle diagram in Fig. 2), respectively. From Eq. 1 we see that, when the roles of a target and a source are reversed (here, the roles of $X$ and $Y$), the difference in redundancy is the opposite of the difference in unique information with respect to the other source (here, $Z$). Similarly, Eq. 2 shows that the difference in synergy is the opposite of the difference in unique information with respect to the other source. Combining these two equalities, we also see that the difference in redundancy is equal to the difference in synergy. Therefore, the equalities impose relationships across some PID atoms appearing in two different diagrams.

These relationships are depicted in Figure  3. The eight PID atoms appearing in the two diagrams can be expressed in terms of only six subatoms, due to the constraints of the form of Eqs. 1 and 2. In particular, to select the smallest nonnegative pieces of information resulting from the constraints, we define:

[TABLE]

The above terms are called the Reversible Shared Information of $X$ and $Y$ considering $Z$ ($RSI(X\overset{\text{Z}}{\leftrightarrow}Y)$; the orange block in Fig. 3), the Reversible Complementary Information of $X$ and $Y$ considering $Z$ ($RCI(X\overset{\text{Z}}{\leftrightarrow}Y)$; the gray block in Fig. 3), and the Reversible Unique Information of $X$ and $Y$ considering $Z$ ($RUI(X\overset{\text{Z}}{\leftrightarrow}Y)$; the magenta block in Fig. 3). The attribute reversible highlights that, when we reverse the roles of target and source between the two variables at the endpoints of the arrow in $RSI$, $RCI$, or $RUI$ (here, $X$ and $Y$), the reversible pieces of information are still included in the same type of PID atom (redundancy, synergy, or unique information with respect to the third variable). For example, the orange block in Fig. 3 indicates a common amount of redundancy in both PID diagrams: as such, $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)$ contributes both to redundant information that $Y$ and $Z$ share about $X$, and to redundant information that $X$ and $Z$ share about $Y$. By construction, these reversible components are symmetric in the reversed variables. Note that, when we reverse the role of two variables, the third variable (here, $Z$) remains a source and is thus put in the middle of our notation in Eqs. 3a,  3b and  3c. We also define the Irreversible Shared Information $IRSI(X\overset{\text{Z}}{\leftarrow}Y)$ between $X$ and $Y$ considering $Z$ (the light blue block in Fig. 3) as follows:

[TABLE]

The attribute irreversible in the above definition indicates that this piece of redundancy is specific to one of the two PIDs alone. More precisely, the uni-directional arrow in $IRSI(X\overset{\text{Z}}{\leftarrow}Y)$ indicates that this piece of information is a part of the redundancy with $X$ as a target, but it is not a part of the redundancy with $Y$ as a target222In this paper, directional arrows never represent any kind of causal directionality: the PID framework is only capable to quantify statistical (correlational) dependencies.. Correspondingly, at least one between $IRSI(X\overset{\text{Z}}{\leftarrow}Y)$ and $IRSI(Y\overset{\text{Z}}{\leftarrow}X)$ is always zero. More generally, $IRSI$ quantifies asymmetries between two different PIDs: for example, when moving from the left to the right PID in Fig. 3, the light blue block $IRSI(X\overset{\text{Z}}{\leftarrow}Y)$ indicates an equivalent amount of information that is lost for the redundancy $SI(X:\{Y;Z\})$ and the synergy $CI(X:\{Y;Z\})$ atoms, and is instead counted as a part of the unique information $UI(Y:\{X\backslash Z\})$ atom. In other words, assuming that the two redundancies are ranked as in Fig. 3, we find that:

[TABLE]

While the coarser Shannon information quantities that are decomposed in both diagrams in Fig. 3, namely $I(X:Y)$ and $I(X:Y|Z)$, are symmetric under swap of $X\leftrightarrow Y$, their PID decompositions (see Eqs. 1 and  2) are not: Eq. 5 show that $IRSI$ quantifies the amount of this asymmetry. More precisely, the PID decompositions of $I(X:Y)$ and $I(X:Y|Z)$ will preserve the $X\leftrightarrow Y$ symmetry if and only if $IRSI(X\overset{\text{Z}}{\leftarrow}Y)=IRSI(Y\overset{\text{Z}}{\leftarrow}X)=0$. Note that, in general, the differences of redundancies, of synergies, and of unique information terms are always constrained by equations such as Eq. 5. Hence, unlike for the reversible measures, we do not need to consider independent notions of irreversible synergy or irreversible unique information.

In summary, the four subatoms in Eqs. 3a,  3b,  3c and  4, together with the two remaining unique information terms (the black dots in Fig. 3), allow us to characterize both PIDs in Fig. 3 and to understand how the PID atoms change when moving from one PID to another. Note that in all cases the blocks indicate amounts of information, but the interpretation of this information depends on the classification of variables as target and sources within each diagram.

3.2 Unveiling the finer structure of the PID framework

So far we have examined the relationships among the PID atoms corresponding to two different perspectives we hold about the system, whereby we reverse the roles of target and source between two variables in the system. We have seen that the PID atoms of different diagrams are not independent, as they are constrained by equations of the type of Eqs. 1 and  2. More specifically, the eight PID atoms of two diagrams can be expressed in terms of only six independent quantities, including reversible and irreversible pieces of information. The next question is how many subatoms we need to describe all the three possible PIDs (see Fig. 2). Since there are six constraints, three equations of the type of Eq. 1 and three equations of the type of Eq. 2, one may be tempted to think that the twelve PID atoms of all three PID diagrams can be expressed in terms of only six independent quantities. However, the six constraints are not independent: this is most easily seen from the symmetry of the co-information measure [31], which is defined as the mutual information of two variables minus their conditional information given the third (e.g., Eq. 1 minus Eq. 2). The co-information is invariant to any permutation of the variables, and this property highlights that only five of the six constraints are linearly independent. Accordingly, we will now detail how seven subatoms are sufficient to describe the whole set of PIDs: we call these subatoms the minimal subatoms’ set of the PID diagrams.

In Fig. 4 we see how the minimal subatoms’ set builds the PID diagrams. We assume, without loss of generality, that $SI(Y:\{X;Z\})\leq SI(X:\{Y;Z\})\leq SI(Z:\{X;Y\})$. Then, we consider the three possible instances of Eq. 5 for the three possible choices of the target variable, and we find that the same ordering also holds for the synergy atoms: $CI(Y:\{X;Z\})\leq CI(X:\{Y;Z\})\leq CI(Z:\{X;Y\})$. This property is related to the invariance of the co-information: indeed, Ref. [10] indicated that the co-information can be expressed as the difference between the redundancy and the synergy within each PID diagram, i.e.

[TABLE]

for any assignment of $X$, $Y$, $Z$, to $i$, $j$, $k$.

These ordering relations are enough to understand the nature of the minimal subatoms’ set: we start with the construction of the three redundancies, which can all be expressed in terms of the smallest $RSI$ and two subsequent increments. In Fig. 4, these correspond respectively to $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)=SI(Y:\{X;Z\})$ (orange block), $IRSI(X\overset{\text{Z}}{\leftarrow}Y)$ (light blue block) and $IRSI(Z\overset{\text{Y}}{\leftarrow}X)$ (yellow block). In parallel, we can construct the three synergies with the smallest $RCI$ and the same increments used for the redundancies. In Fig.  4, these correspond respectively to $RCI(X\overset{\text{Z}}{\leftrightarrow}Y)=CI(Y:\{X;Z\})$ (gray block) and the same two $IRSI$ used before. To construct the unique information atoms, it is sufficient to further consider the three independent $RUI$ defined by taking all possible permutations of $X$, $Y$ and $Z$ in Eq. 3c. In Fig. 4, these correspond to $RUI(X\overset{\text{Z}}{\leftrightarrow}Y)=UI(X:\{Y\backslash Z\})$ (magenta block), $RUI(X\overset{\text{Y}}{\leftrightarrow}Z)=UI(Z:\{X\backslash Y\})$ (brown block), and $RUI(Y\overset{\text{X}}{\leftrightarrow}Z)=UI(Z:\{Y\backslash X\})$ (blue block). We thus see that, in total, seven minimal subatoms are enough to build the three PID diagrams of any system. Among these seven building blocks, five are reversible pieces of information, i.e. they contribute to the the same kind of PID atom across different PID diagrams; the other two are irreversible pieces of information, that contribute to different kinds of PID atom across different diagrams. The complete minimal set can only be determined when all three PIDs are jointly considered and compared: as shown in Fig.3, pairwise PIDs’ comparisons can at most distinguish two subatoms in any redundancy (or synergy), while the three-wise PIDs’ comparison discussed above allowed us to discern three subatoms in $SI(Z:\{X;Y\})$ (and $CI(Z:\{X;Y\})$; see also Fig.4).

Importantly, while the definition of a single PID lattice relies on the specific perspective adopted on the system, which labels two variables as the sources and one variable as the target, the decomposition in Fig. 4 is invariant with respect to the classification of the variables. As described above, it only relies on computing all three PID diagrams and then using the ordering relations of the atoms, without any need to classify the variables a priori. As illustrated in Fig. 4, the decomposition of the mutual information and conditional mutual information quantities in terms of the subatoms is also independent of the PID adopted. Our invariant minimal set in Fig. 4 thus extends the descriptive power of the PID framework beyond the limitations that were intrinsic to considering an individual PID diagram. In the next sections, we will show how the invariant minimal set can be used to identify the part of the redundant information about a target that specifically arises from the mutual information between the sources (the source redundancy), and to decompose the total entropy of any trivariate system.

Remarkably, the decomposition in Fig. 4 does not rely on any extension of Williams and Beer’s axioms. We unveiled finer structure underlying the PID lattices just by considering more PID lattices at a time and comparing PID atoms across different lattices. We further remark that the decomposition in Fig. 4 does not rely in any respect on the specific definition of the PID measures that is used to calculate the PID atoms: it only relies on the axiomatic PID construction presented in Ref. [10].

4 Quantifying source redundancy

The structure of the three PID diagrams that was unveiled with the construction in Fig. 4 enables a finer characterization of the modes of information distribution among three variables than what has previously been possible. In particular, we will now address the open problem of quantifying the source redundancy, i.e. the part of the redundancy that ’must already manifest itself in the mutual information between the sources’ [22]. Consider for example the redundancy $SI(X:\{Y;Z\})$ in Fig. 4: it is composed by $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)$ (orange block) and $IRSI(X\overset{\text{Z}}{\leftarrow}Y)$ (light blue block). We can check which of these subatoms are shared with the mutual information of the sources $I(Y:Z)$. To do this, we have to move from the middle PID diagram in Fig. 4, that contains $SI(X:\{Y;Z\})$, to any of the other two diagrams, that both contain $I(Y:Z)$. Consistently, in these other two diagrams $I(Y:Z)$ is composed by the same four subatoms (the orange, the light blue, the yellow and the blue block), and the only difference across diagrams is that these subatoms are differently distributed between unique information and redundancy PID atoms. In particular, we can see that both the orange and the light blue block which make up $SI(X:\{Y;Z\})$ are contained in $I(Y:Z)$. Thus, whenever any of them is nonzero, we know at the same time that $Y$ and $Z$ share some information about $X$ (i.e., $SI(X:\{Y;Z\})>0$) and that there are correlations between $Y$ and $Z$ (i.e., $I(Y:Z)>0$). Accordingly, in the scenario of Fig. 4 the entire redundancy $SI(X:\{Y;Z\})$ is explained by the mutual information of the sources: all the redundant information that $Y$ and $Z$ share about $X$ arises from the correlations between $Y$ and $Z$.

If we then consider the redundancy $SI(Y:\{X;Z\})$, that coincides with the orange block, we also find that it is totally explained in terms of the mutual information between the corresponding sources $I(X:Z)$, which indeed contains an orange block. However, if we consider the third redundancy $SI(Z:\{X;Y\})$, that is composed by an orange, a light blue and a yellow block, we find that only the orange and the light blue block contribute to $I(X:Y)$, while the yellow does not. This means that if $IRSI(Z\overset{\text{Y}}{\leftarrow}X)>0$ (yellow block), then $X$ and $Y$ can share information about $Z$ (i.e., $SI(Z:\{X;Y\})>0$) even if there is no mutual information between the sources $X$ and $Y$ (i.e., $I(X:Y)=0$). Following this reasoning, we define the source redundancy that two sources $S_{1}$ and $S_{2}$ share about a target $T$ as:

[TABLE]

One can easily verify that Eq. 7 identifies the blocks that belong to both $SI(T,\{S_{1},S_{2}\})$ and $I(S_{1}:S_{2})$ in Fig.  4, for any choice of sources and target (for instance, $T=Z$, $S_{1}=X$ and $S_{2}=Y$). This definition can be justified as follows: each $RSI$ measure in Eq. 7 compares $SI(T:\{S_{1};S_{2}\})$ with one of the other two redundancies that are contained in the mutual information between the sources $I(S_{1}:S_{2})$ (namely, $SI(S_{1}:\{S_{2};T\})$ and $SI(S_{2}:\{S_{1};T\})$). Some of the subatoms included in $I(S_{1}:S_{2})$ are contained in one of these two redundancies, but not in the other, as they move to the unique information mode when we change PID. Therefore, by taking the maximum in Eq. 7 we ensure that $SR(T:\{S_{1};S_{2}\})$ captures all the common subatoms of $I(S_{1}:S_{2})$ and $SI(T:\{S_{1};S_{2}\})$. In a complementary way, we can define the non-source redundancy that two sources share about a target:

[TABLE]

4.1 The difference between source and non-source redundancy

Eqs. 7 and  8 show how we can split the redundant information that two sources share about a target into two nonnegative information components: when the source-redundancy $SR$ is larger than zero there are also correlations between the sources, while the non-source redundancy $NSR$ can be larger than zero even when the sources are independent. $SR$ is thus seen to quantify the pairwise correlations between the sources that also produce redundant information about the target: this discussion is pictorially summarized in Fig. 5. In particular, the source redundancy $SR$ is clearly upper-bounded by the mutual information between the sources, i.e.

[TABLE]

On the other hand, $NSR$ does not arise from the pairwise correlations between the sources: let us calculate $NSR$ in a paradigmatic example that was proposed in Ref. [22] to remark the subtle possibility that two statistically independent variables can share information about a third variable. Suppose that $Y$ and $Z$ are uniform binary random variables, with Y\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z, and $X$ is deterministically fixed by the relationship $X=Y\wedge Z$. Here, $SI(X:\{Y;Z\})\approx 0.311$ bit (according to different measures of redundancy  [22, 11]) even if $I(Y:Z)=0$. Indeed, from our definitions in Eqs.  7 and  8 we find that, since $SI(Y:\{X;Z\}),SI(Z:\{X;Y\})\leq I(Y:Z)=0$, here $NSR(X:\{Y;Z\})=SI(X:\{Y;Z\})>0$ even though $I(Y:Z)=0$. We will comment more extensively on this instructive example in Section  6.

Interestingly, the non-source redundancy $NSR$ is a part of the redundancy that is related to the synergy of the same PID diagram. Indeed, two of the three possible $NSR$ defined in Eq. 8 are always zero, and the third can be larger than zero if and only if the yellow block in Fig. 4 is larger than zero. From Fig. 4 and Fig. 5 we can thus see that, whenever we find positive non-source redundancy in a PID diagram, the same amount of information (the yellow block) is also present in the synergy of that diagram. Thus, while there is source redundancy if and only if there is mutual information between the sources, the existence of non-source redundancy is a sufficient (though not necessary) condition for the existence of synergy. We can thus interpret $NSR$ as redundant information about the target that implies that the sources carry synergistic information about the target: we give a graphical characterization of $NSR$ in Fig. 5.

In the specific examples considered in [22], where the underlying causal structure of the system is such that the sources always generate the target, the non-source redundancy can indeed be associated with the notion of ’mechanistic redundancy’ that was introduced in that work: the causal mechanisms connecting the target with the sources induce a non-zero $NSR$ that contributes to the redundancy independently of the correlations between the sources. In general, since the causal structure of the analyzed system is unknown, it is impossible to quantify ’mechanistic redundancy’ with statistical measures, while it is always possible to quantify and interpret the non-source redundancy as described in Section 4.

In section 6 we will examine concrete examples to show how our definitions of source and non-source redundancy refine the information-theoretic description of trivariate systems, as they quantify qualitatively different ways that two variables can share information about a third.

We conclude this Section with more general comments about our quantification of source redundancy. We note that the arguments used to define the source redundancy in Sec. 4 can be equally used to study common or exclusive information components of other PID terms. For example, we can identify the magenta subatom as the component of the mutual information between the sources $X$ and $Y$ that cannot be related to their redundant information about $Z$. Similarly, we could consider which part of a synergy is related to the conditional mutual information between the sources.

5 Decomposing the joint entropy of a trivariate system

Understanding how information is distributed in trivariate systems should also provide a descriptive allotment of all parts of the joint entropy $H(X,Y,Z)$  [21, 30]. For comparison, Shannon’s mutual information enables a semantic decomposition of the bivariate entropy $H(X,Y)$ in terms of univariate conditional entropies and $I(X:Y)$, that quantifies shared fluctuations (or covariations) between the two variables [30]:

[TABLE]

However, in spite of recent efforts [30], a univocal descriptive decomposition of the trivariate entropy $H(X,Y,Z)$ is still missing to date. Since the PID axioms in Ref. [10] decompose mutual information quantities, one might hope that the PID atoms could also provide a descriptive entropy decomposition. Yet, at the beginning of Section 3, we pointed out that a single PID lattice does not include the mutual information between the sources and their conditional mutual information given the target: this suggests that a single PID lattice cannot in general contain the full $H(X,Y,Z)$. More concretely, Ref. [21] has recently suggested precise examples of trivariate dependencies where a single PID lattice cannot account for, and thus describe the parts of, the full $H(X,Y,Z)$.

These examples compare the dyadic and triadic dependencies described in Fig. 6. Both kinds of dependencies underlie common modes of information sharing among three and more variables [21]. Ref. [21] remarked that the atoms of a single PID diagram are indeed able to distinguish between dyadic and triadic dependencies, but that such atoms only sum up to two of the three bits of the full $H(X,Y,Z)$.

We will now show how the missing allotment of the third bit of entropy in those systems is not due to intrinsic limitations of the PID axioms, but just to the limitations of considering a single PID diagram at a time — the common practice in the literature so far. More generally, we will be able to allot and describe the full entropy $H(X,Y,Z)$ of any trivariate system in terms of the novel finer structure unveiled in Section 3.

5.1 The finer structure of the entropy $H(X,Y,Z)$

The minimal subatoms’ set that we illustrated in Fig. 4 allowed us to decompose all three PID lattices of a generic system. However, to fully describe the distribution of information in trivariate systems, we also wish to find a generalization of Eq. 10 to the trivariate case, i.e. to decompose the full trivariate entropy $H(X,Y,Z)$ in terms of univariate conditional entropies and PID quantities. With this goal in mind, we first subtract from $H(X,Y,Z)$ the terms which describe statistical fluctuations of only one variable (conditioned on the other two). The sum of these terms was indicated as $H_{(1)}$ in Ref. [30], and there quantified as

[TABLE]

This subtraction is useful because $H_{(1)}$ is a part of the total entropy which does not overlap with any of the 12 PID atoms in Fig. 2. The remaining entropy $H(X,Y,Z)-H_{(1)}$ was defined as the dual total correlation in Ref. [32] and recently considered in Ref. [30]:

[TABLE]

$DTC$ quantifies joint statistical fluctuations of more than one variable in the system. A simple calculation yields

[TABLE]

which is manifestly invariant under permutations of $X$, $Y$ and $Z$, and shows that $DTC$ can be written as a sum of some of the $12$ PID atoms. For example, expressing the co-information as the difference $I(X:Z)-I(X:Z|Y)$, we can arbitrarily use the four atoms from the left-most diagram in Fig. 2 to decompose the sum $I(X:Y|Z)+I(X:Z)$ and then add $UI(Y:\{Z\backslash X\})+CI(Y:\{X;Z\})$ from the middle diagram to decompose $I(Y:Z|X)$. If we then plug this expression of $DTC$ in Eq. 12, we achieve a decomposition of the full entropy of the system in terms of $H_{(1)}$ and PID quantities:

[TABLE]

which provides a nonnegative decomposition of the total entropy of any trivariate system. However, this decomposition is not unique, since the co-information can be expressed in terms of different pairs of conditional and unconditional mutual informations, according to Eq. 6. This arbitrariness strongly limits the descriptive power of this kind of entropy decompositions, because the PID atoms on the RHS can only be interpreted within individual PID perspectives.

To address this issue, we construct a less arbitrary entropy decomposition by using the invariant minimal subatoms’ set that was presented in Fig. 4: importantly, that set can be interpreted without specifying an individual, and thus partial, PID point of view that we hold about the system.

Thus, after we name the variables of the system such that $SI(Y:\{X;Z\})\leq SI(X:\{Y;Z\})\leq SI(Z:\{X;Y\})$, we express the coarser PID atoms in Eq. 5.1 in terms of the minimal set to obtain:

[TABLE]

Unlike Eq. 5.1, the entropy decomposition expressed in Eq. 15 and illustrated in Fig. 7 fully describes the distribution of information in trivariate systems without the need of a specific perspective about the system. Importantly, this decomposition is unique: even though the co-information can be expressed in different ways in terms of conditional and unconditional mutual informations, in terms of the subatoms it is uniquely represented as the orange block minus the gray block (see Fig. 7). Similarly, the conditional mutual information terms of Eq. 13 are composed by the same blocks independently of the PID, as highlighted in Fig. 4.

5.2 Describing $H(X,Y,Z)$ for dyadic and triadic systems

To test the usefulness of the finer entropy decomposition in Eq. 15, we now compute its terms for the dyadic and the triadic dependencies considered in Ref. [21] and defined in Figure 6. In both cases $H_{(1)}=0$. For the dyadic system, there are only three positive quantities in the minimal set: the three reversible unique informations $RUI(X\overset{\text{Y}}{\leftrightarrow}Z)=RUI(Y\overset{\text{X}}{\leftrightarrow}Z)=RUI(X\overset{\text{Z}}{\leftrightarrow}Y)=1$ bit. For the triadic system, there are only two positive quantities in the minimal set: $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)=RCI(X\overset{\text{Z}}{\leftrightarrow}Y)=1$ bit, but $RCI(X\overset{\text{Z}}{\leftrightarrow}Y)$ is counted twice in the $DTC$. We illustrate the resulting entropy decompositions, according to Eq. 15, in Fig. 8.

The decompositions in Fig. 8 enable a clear interpretation of how information is finely distributed within dyadic and triadic dependencies. The three bits of the total $H(X,Y,Z)$ in the dyadic system are seen to be distributed equally among unique information modes: each variable contains 1 bit of unique information with respect to the second variable about the third variable. Further, these unique information terms are all reversible, which reflects the symmetry of the system under pairwise swapping of the variables. This description provides a simple and accurate summary of the total entropy of the dyadic system, which matches the dependency structure illustrated in Fig. 6(a).

The three bits of the total $H(X,Y,Z)$ in the triadic system consist of one bit of the smallest reversible redundancy and two bit of the smallest reversible synergy, since the latter appears twice in $H(X,Y,Z)$. Again, the reversible nature of these pieces of information reflects the symmetry of the system under pairwise swapping of the variables. Further, the bit of reversible redundancy represents the bit of information that is redundantly available to all three variables, while the two bit of reversible synergy are due to the three-wise XOR structure (see Fig. 6(b)). Why does the XOR structure provide two bits of synergistic information? Because if $X=Y\oplus Z$ then the only positive quantity in the set of subatoms in Fig. 4b is the smallest $RCI$, which however appears twice in the entropy $H(X,Y,Z)$. Importantly, these two bits of synergy do not come from the same PID diagram: our entropy decomposition in Eq. 15 could account for both bits only because it fundamentally relies on cross-comparisons between different PID diagrams, as illustrated in Fig. 4.

6 Applications of the finer structure of the PID framework

The aim of this Section is to show the additional insights that the finer structure of the PID framework, unveiled in Section  3.2 and Fig. 4, can bring to the analysis of trivariate systems. We examine paradigmatic examples of trivariate systems and calculate the novel PID quantities of source and non-source redundancy that we described in Sec. 4. Most of these examples have been considered in the literature  [22, 24, 11, 23, 30] to validate the definitions, or to suggest interpretations, of the PID atoms. We also discuss how $SR$ matches the notion of source redundancy introduced in Ref. [22] and discussed in Ref. [20]. Finally, we suggest and motivate a practical interpretation of the reversible redundancy subatom $RSI$.

Even though our definitions of the minimal subatoms’ set in Fig. 4 only rely on Williams and Beer’s axioms, some of the examples below will require a specific definition of the PID atoms that goes beyond those axioms. In those cases, our computations rely on the definitions of PID that were proposed in Ref. [11], which are the most widely accepted in the literature for trivariate systems. Where numerical computations of the PID atoms are involved, they have been performed with a software package that will be publicly released upon publication of the present work.

6.1 Computing source and non-source redundancy

6.1.1 Copying — the redundancy arises entirely from source correlations

Consider a system where $Y$ and $Z$ are random binary variables that are correlated according to a control parameter $\lambda$ [22]. For example, consider a uniform binary random variable $W$ that ’drives’ both $Y$ and $Z$ with the same strength $\lambda$. More precisely, $p(y|w)=\lambda/2+(1-\lambda)\delta_{yw}$ and $p(z|w)=\lambda/2+(1-\lambda)\delta_{zw}$ [22]. This system is completed by taking $X=(Y,Z)$, i.e. a two-bit random variable that reproduces faithfully the joint outcomes of the generating variables $Y,Z$.

We consider the inputs $Y$ and $Z$ as the PID sources and the output $X$ as the PID target, thus selecting the left-most PID diagram in Fig. 2. Fig. 9(a) shows our calculations of the full redundancy $SI(X:\{Y;Z\})$, the source redundancy $SR(X:\{Y;Z\})$ and the non-source redundancy $NSR(X:\{Y;Z\})$, based on the definitions in Ref. [11]. The parameter $\lambda$ is varied between $\lambda=0$, corresponding to $Y=Z$, and $\lambda=1$, corresponding to Y\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z. Since $NSR(X:\{Y;Z\})=0$ for any $0\leq\lambda\leq 1$, we interpret that all the redundancy $SI(X:\{Y;Z\})$ arises from the correlations between $Y$ and $Z$ (which are tuned with $\lambda$). This is indeed compatible with the discussion in Ref. [22], where the authors argued that in the ’copying’ example the entire redundancy should be already apparent in the sources.

6.1.2 AND gate: the redundancy is not entirely related to source correlations

Consider a system where the correlations between two binary random variables, the inputs $Y$ and $Z$, are described by the control parameter $\lambda$ as in 6.1.1, but the output $X$ is determined by the $AND$ function as $X=Y\wedge Z$ [22]. As the causal structure of the system would suggest, we consider the inputs $Y$ and $Z$ as the PID sources and the output $X$ as the PID target, thus selecting the left-most PID diagram in Fig. 2. Fig. 9(b) shows our calculations of the full redundancy $SI(X:\{Y;Z\})$, the source redundancy $SR(X:\{Y;Z\})$ and the non-source redundancy $NSR(X:\{Y;Z\})$, based on the definitions in Ref. [11].

$SR$ and $NSR$ now show a non-trivial behavior as a function of the $Y-Z$ correlation parameter $\lambda$. If $\lambda=0$ and thus $Y=Z$, the full redundancy is made up entirely of source redundancy — trivially, both $SI(X:\{Y;Z\})$ and $SR(X:\{Y;Z\})$ equal the mutual information $I(Y:Z)=H(Y)=1$ bit. When $\lambda$ increases, the full redundancy decreases monotonically to its minimum value of $\approx 0.311$ bit for $\lambda=1$ (when Y\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z). Importantly, $SR(X:\{Y;Z\})$ decreases monotonically as a function of $\lambda$ to its minimum value of zero bit when $\lambda=1$: this behavior is indeed expected from a measure that quantifies correlations between the sources that also produce redundant information about the target (see Section 4). On the other hand, $NSR(X:\{Y;Z\})>0$ for $\lambda>0$, and it increases as a function of $\lambda$. When $\lambda=1$, i.e. when Y\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z, $NSR$ corresponds to the full redundancy. This is compatible with our description of non-source redundancy (see Section  4) as redundancy that is not related to the source correlations; indeed, $NSR>0$ implies that the sources also carry synergistic information about the target (here, due to the relationship $X=Y\wedge Z$). $NSR$ thus also quantifies the notion of mechanistic redundancy that was introduced in Ref. [22] with reference to this scenario.

6.1.3 Dice sum: tuning irreversible redundancy

Consider a system where $Y$ and $Z$ are two uniform random variables, each representing the outcome of a die throw [22]. A parameter $\lambda$ controls the correlations between the two dice: $p(y,z)=\lambda/36+(1-\lambda)/6\hskip 2.84526pt\delta_{yz}$. Thus, for $\lambda=0$ the dice throws always match, for $\lambda=1$ the outcomes are completely independent. Further, the output $X$ combines each pair $(y,z)$ of input outcomes as $x=y+\alpha$ $z$, where $\alpha\in\{1,2,3,4,5,6\}$. This example was suggested in Ref. [22] specifically to point out the conceptual difficulties in quantifying the interplay between the redundancy and the source correlations, that we addressed with the identification of $SR$ and $NSR$ (see Section  4).

Harder et al. calculated redundancy by using their own proposed measure $I_{\text{red}}$, which also abides by the PID axioms in Ref. [10] but is different than the measure $SI(X:\{Y;Z\})$ introduced in Ref. [11]. However, Ref. [11] also calculated $SI(X:\{Y;Z\})$ for this example: they showed that the differences between $SI(X:\{Y;Z\})$ and $I_{\text{red}}$ in this example are only quantitative (and there is no difference at all for some values of $\alpha$), while the qualitative behaviour of both measures as a function of $\lambda$ is very similar.

Fig. 10 shows our calculations of the full redundancy $SI(X:\{Y;Z\})$, the source redundancy $SR(X:\{Y;Z\})$ and the non-source redundancy $NSR(X:\{Y;Z\})$, based on the definitions in Ref. [11]. We display the two ’extreme’ cases $\alpha=1$, $\alpha=6$ for illustration.

With $\alpha=6$, $X$ is isomorphic to the joint variable $(Y,Z)$: for any $\lambda$, this implies on one hand $SI(X:\{Y;Z\})=SI((Y;Z):\{Y;Z\})=I(Y:Z)$, and on the other $UI(Y:\{Z\backslash X\})=0$ and thus $SI(Y:\{X;Z\})=I(Y:Z)$ (see Eq. 1). According to Eq. 7, we thus find that the source redundancy $SR(X:\{Y;Z\})$ saturates, for any $\lambda$, the general inequality in Eq. 9: all correlations between the inputs $Y,Z$ also produce redundant information about $X$ (see Section  4). Further, according to Eq. 8, we find $NSR(X:\{Y;Z\})=0$ for any $\lambda$ (see Fig. 10(b)): we thus interpret that all the redundancy $SI(X:\{Y;Z\})$ arises from correlations between the inputs. Instead, if we fix $\lambda$ and decrease $\alpha$, the two inputs $Y,Z$ are more and more symmetrically combined in the output $X$: with $\alpha=1$, the pieces of information respectively carried by each input about the output overlap maximally. Correspondingly, the full redundancy $SI(X:\{Y;Z\})$ increases  [22]. However, keeping $\lambda$ fixed does not change the inputs’ correlations. Thus, we expect that the relative contribution of the inputs’ correlations to the full redundancy $SI(X:\{Y;Z\})$ should decrease proportionally. Indeed, in Fig. 10(a) we find $NSR(X:\{Y;Z\})>0$ for $\lambda>0$, which signals that a part of $SI(X:\{Y;Z\})$ is not related to the inputs’ correlations.

We finally note that also in this paradigmatic example the splitting of the redundancy into $SR$ and $NSR$ addresses the challenge of separating the two kinds of redundancy outlined in Ref. [22].

6.1.4 Trivariate jointly Gaussian systems

Barrett considered in detail the application of the PID to trivariate jointly Gaussian systems $(X,Y,Z)$ in which the target is univariate [23]: he showed that several specific proposals for calculating the PID atoms all converge, in this case, to the same following measure of redundancy:

[TABLE]

We note that Eq. 16 highlights the interesting property that, in trivariate Gaussian systems with a univariate target, the redundancy is as large as it can be, since it saturates the general inequalities $SI(X:\{Y;Z\})\leq I(X:Y),I(X:Z)$.

Direct application of our definitions in Eqs. 7 and 8 to such systems yields:

[TABLE]

Thus, we find that in these systems the source redundancy is also as large as it can be, since it also saturates the general inequalities $SR(X:\{Y;Z\})\leq I(X:Y),I(X:Z),I(Y:Z)$ (which follow immediately from its definition in Eq. 7). Further, combining Eq. 16 with Eq. 17 gives:

[TABLE]

This identification of source redundancy, which quantifies pairwise correlations between the sources that also produce redundant information about the target (see Section 4), provides more insight about the distribution of information in Gaussian systems. Indeed, the property that source redundancy is maximal indicates that the correlations between any pair of source variables (for example, $\{Y;Z\}$ as considered above) produces as much redundant information as possible about the corresponding target ($X$ as considered above). Accordingly, when $I(Y:Z)<SI(X:\{Y;Z\})$, the redundancy also includes some non-source redundancy $NSR(X:\{Y;Z\})>0$ that implies the existence of synergy, i.e. $CI(X:\{Y;Z\})>0$.

6.2 $RSI$ quantifies information between two variables that also passes monotonically through the third

In this section we discuss a practical interpretation of the reversible redundancy subatom $RSI$ defined in Eq.3a. We note that $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)$ appears both in $SI(X:\{Y;Z\})$ and in $SI(Y:\{X;Z\})$, i.e. it quantifies a common amount of information that $Z$ shares with each of the two other variables about the third variable. We shorten this description into the statement that $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)$ quantifies ’information between $X$ and $Y$ that also passes through $Z$’. Reversible redundancy is further characterized by the following Proposition:

Proposition 1**.**

$RSI(X\overset{\text{(Z,Z')}}{\leftrightarrow}Y)$* $\geq$ $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)$.*

Proof.

Both $SI(X:\{Y;(Z,Z^{\prime})\})\geq SI(X:\{Y;Z\})$ and $SI(Y:\{X;(Z,Z^{\prime})\})\geq SI(Y:\{X;Z\})$  [3]. Thus, $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)=\min[SI(X:\{Y;Z\}),SI(Y:\{X;Z\})]\leq\min[SI(X:\{Y;(Z,Z^{\prime})\}),SI(Y:\{X;(Z,Z^{\prime})\})]=RSI(X\overset{\text{(Z,Z')}}{\leftrightarrow}Y)$. ∎

Indeed, the fact that $RSI$ always increases whenever we expand the middle variable corresponds to the increased capacity of the entropy of the middle variable to host information between the endpoint variables. We discuss this interpretation of $RSI$ by examining several examples, where we can motivate a priori our expectations about this novel mode of information sharing.

6.2.1 Markov chains

Consider the most generic Markov chain $X\rightarrow Z\rightarrow Y$, defined by the property that $p(x,y|z)=p(x|z)p(y|z)$, i.e. that $X$ and $Y$ are conditionally independent given $Z$. The Markov structure allows us to formulate a clear a priori expectation about the amount of information between the endpoints of the chain, $X$ and $Y$, that also ’passes through the middle variable $Z$’: this information should clearly equal $I(X:Y)$, because whatever information is established between $X$ and $Y$ must pass through $Z$.

Indeed, the Markov property $I(X:Y|Z)=0$ implies, as we see immediately from Fig. 2, that $SI(X:\{Y;Z\})=SI(Y:\{X;Z\})=I(X:Y)$. By virtue of Eqs. 3a and 4, in accordance with our expectation, we find

[TABLE]

which holds true independently of the marginal distributions of $X$, $Y$ and $Z$. Notably, the symmetry of $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)$ under swap of the endpoint variables $X\leftrightarrow Y$ is also compatible with the property that $X\rightarrow Z\rightarrow Y$ is a Markov chain if and only if $Y\rightarrow Z\rightarrow X$ is a Markov chain. In words, whatever information flows from $X$ through $Z$ to $Y$ equals the information flowing from $Y$ through $Z$ to $X$.

More generally, the Markov property $I(X:Y|Z)=0$ implies that $RCI(Y\overset{\text{Z}}{\leftrightarrow}X)=IRSI(X\overset{\text{Z}}{\leftarrow}Y)=RUI(X\overset{\text{Z}}{\leftrightarrow}Y)=0$: thus, only four of the seven subatoms of the minimal set in Fig. 4 can be larger than zero. The three PIDs of the system, decomposed with the minimal set, are shown in Fig. 11. In particular, we see that also $RSI(X\overset{\text{Y}}{\leftrightarrow}Z)=I(X:Y)$, thus matching our expectation that in the Markov chain $X\rightarrow Z\rightarrow Y$ the information between $X$ and $Z$ that also passes through $Y$ still equals $I(X:Y)$. We finally note that none of the results regarding Markov chains depends on specific definitions of the PID atoms: they were derived only on the basis of the PID axioms in Ref. [10].

6.2.2 Two parallel communication channels

Consider five binary uniform random variables $X_{1},X_{2},Y_{1},Y_{2},Z$ with three parameters $0\leq\lambda_{1},\lambda_{2},\lambda_{3}\leq 1$ controlling the correlations $X_{1}\overset{\lambda_{1}}{\leftrightarrow}Z\overset{\lambda_{2}}{\leftrightarrow}Y_{1}$, $X_{2}\overset{\lambda_{3}}{\leftrightarrow}Y_{2}$ (in the same way $\lambda$ controls the $Y\overset{\lambda}{\leftrightarrow}Z$ correlations in  6.1.1). We consider the trivariate system ($X,Y,Z$) with $X=(X_{1},X_{2})$ and $Y=(Y_{1},Y_{2})$ (see Fig. 12). We intuitively expect that the information between $X$ and $Y$ that also passes through $Z$, in this case, should equal $I(X_{1}:Y_{1})$, which in general will be smaller than $I(X:Y)=I(X_{1}:Y_{1})+I(X_{2}:Y_{2})$. Indeed, we computed the PID atoms with the definitions in Ref. [11] over a fine discretization of the $(\lambda_{1},\lambda_{2},\lambda_{3})$-space and we always found that $RSI(X\overset{\text{Y}}{\leftrightarrow}Z)=I(X_{1}:Y_{1})$.

6.2.3 Other examples

To further describe the interpretation of $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)$ as information between $X$ and $Y$ that also passes through $Z$, we here reconsider the examples, among those discussed before Section 6.2, where we can formulate intuitive expectations about this information sharing mode. In the dyadic system described in Fig. 6(a), our expectation is that there should be no information between two variables that also passes through the other: indeed, $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)=0$ in this case. Instead, in the triadic system described in Fig. 6(b), we expect that the information between two variables that also passes through the other should equal the information that is shared among all three variables, which amounts to 1 bit. Indeed, we find $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)=1$ bit. In the ’copying’ example in Section  6.1.1 $X=(Y,Z)$, thus we expect that $I(Y:Z)$ corresponds to information between $Y,Z$ that also passes through $X$, but also to information between $X$ and $Z$ that also passes through $Y$. Indeed, $RSI(X\overset{\text{Z}}{\leftrightarrow}Y)=RSI(X\overset{\text{Y}}{\leftrightarrow}Z)=I(Y:Z)$. We remark that all the values of $RSI$ in these examples depend on the specific definitions of the PID atoms in Ref. [11].

7 Discussion

The Partial Information Decomposition (PID) pioneered by Williams and Beer has provided an elegant construction to quantify, with information theory, different ways two stochastic variables can carry information about a third variable [10]. In particular, it has enabled consistent definitions of synergy, redundancy and unique information components among three stochastic variables [11]. More generally, it has generated considerable interest as it addresses the difficult yet practically important problem of extending information theory beyond the classic bivariate measures of Shannon to fully characterize multivariate dependencies.

However, the axiomatic PID construction, as originally formulated by Williams and Beer, fundamentally relied on the possibly arbitrary classification of the three variables into sources and target, and this partial perspective prevented a complete and general information-theoretic description of trivariate systems. More specifically, the original PID framework could not quantify some important modes of information sharing, such as source redundancy [22], and could not allot the full joint entropy of some important trivariate systems [21]. The work presented here addresses these issues by extending the original PID framework in two respects. First, we decomposed the original PID atoms in terms of finer information subatoms with a well defined interpretation that is invariant to different variables’ classifications. Then, we constructed an extended framework to completely decompose the distribution of information within any trivariate system.

Importantly, our formulation did not require the addition of further axioms to the original PID construction. We proposed that distinct PIDs for the same system, corresponding to different target selections, should be evaluated and then compared to identify how the decomposition of information changes across different perspectives. More specifically, we identified reversible pieces of information ($RSI$, $RUI$, $RCI$) that contribute to the same kind of PID atom if we reverse the roles of target and source between two variables. The complementary subatomic components of the PID lattices are the irreversible pieces of information ($IRSI$), that contribute to different kinds of PID atom for different target selections. These subatoms thus measure asymmetries between different decompositions of Shannon quantities pertaining to two different PIDs of the same system, and such asymmetries reveal the additional detail with which the PID atoms assess trivariate dependencies as compared to the coarser and more symmetric Shannon information quantities.

The crucial result of this approach was unveiling the finer structure underlying the PID lattices: we showed that an invariant minimal set of seven information subatoms is sufficient to decompose the three PIDs of any trivariate system. In the remainder of this section, possible uses of these subatoms and their implications for the understanding of systems of three variables are discussed.

7.1 Use of the subatoms to map the distribution of information in trivariate systems

Our minimal subatoms’ set was first used to characterize more finely the distribution of information among three variables. We clarified the interplay between the redundant information shared between two variables $A$ and $B$ about a third variable $C$, on one side, and the correlations between $A$ and $B$, on the other. We decomposed the redundancy into the sum of source-redundancy $SR$, which quantifies the part of the redundancy which arises from the pairwise correlations between the sources $A$ and $B$, and non-source redundancy $NSR$, which can be larger than zero even if the sources $A$ and $B$ are statistically independent. Interestingly, we found that $NSR$ quantifies the part of the redundancy which implies that $A$ and $B$ also carry synergistic information about $C$. The separation of these qualitatively different components of redundancy promises to be useful in the analysis of any complex system where several inputs are combined to produce an output [20].

Then, we used our minimal subatoms’ set to extend the descriptive power of the PID framework in the analysis of any trivariate system. We constructed a general, unique, and nonnegative decomposition of the joint entropy $H(X,Y,Z)$ in terms of information-theoretic components that can be clearly interpreted without arbitrary variable classifications. This construction parallels the decomposition of the bivariate entropy $H(X,Y)$ in terms of Shannon’s mutual information $I(X:Y)$. We demonstrated the descriptive power of this approach by decomposing the complex distribution of information in dyadic and triadic systems, which was shown not to be possible within the original PID framework [21].

We gave practical examples of how the finer structure underlying the PID atoms provides more insight into the distribution of information within important and well-studied trivariate systems. In this spirit, we put forward a practical interpretation of the reversible redundancy $RSI$, and future work will address additional interpretations of the components of the minimal subatoms’ set.

7.2 Possible extensions of the formalism to multivariate systems with many sources

The insights that derive from our extension of the PID framework suggest that the PID lattices could also be useful to characterize the statistical dependencies in multivariate systems with more than two sources. Our approach does not rely on the adoption of specific PID measures, but only on the axiomatic construction of the PID lattice. Thus, it can be immediately extended to the multivariate case by embedding trivariate lattices within larger systems’ lattices [12]: a further breakdown of the minimal subatoms’ set could be obtained if the current system were embedded as part of a bigger system. Further, when systems with more than two sources are considered, the definition of source redundancy might be extended as to determine which subatoms of a redundancy can be explained by dependencies among the sources (for example, by replacing the mutual information between two sources with a measure of the overall dependencies among all sources, such as the total correlation introduced in Ref. [33]). More generally, the idea of comparing different PID diagrams that partially decompose the same information can also be generalized to identify finer structure underlying higher-order PID lattices with different numbers of variables. These identifications might also help addressing specific questions about the distribution of information in complex multivariate systems.

7.3 Potential implications for systems biology and systems neuroscience

A common problem in system biology is to characterize how the function of the whole biological system is shaped by the dependencies among its many constituent biological variables. In many cases, ranging from gene regulatory networks [34] to metabolic pathways [14] and to systems neuroscience [35, 36, 37], an information-theoretic decomposition of how information is distributed and processed within different parts of the system would allow a model-free characterization of these dependencies. The work discussed here can be used to shed more light on these issues by allowing to tease apart qualitatively different modes of interaction, as a first necessary step to understanding the causal structure of the observed phenomena [38].

In systems neuroscience, the decomposition introduced here may be important for studying specific and timely questions about neural information processing. This work can contribute to the study of neural population coding, that is the study of how the concerted activity of many neurons encodes information about ecologically relevant variables such as sensory stimuli [39, 37]. In particular, a key characteristic of a neural population code is the degree to which pairwise or higher-order cross-neuron statistical dependencies are used by the brain to encode and process information, in a potentially redundant or synergistic way, across neurons [7, 8, 9] and across time [40]. Our work is also of potential relevance to study another hot issue in neuroscience, that is the relevance of the information about sensory variables carried by neural activity for perception [19, 41]. This is a crucial issue to resolve the diatribe about the nature of the neural code, that is the set of symbols used by neurons to encode information and produce brain function [42, 43, 44]. Addressing this problem requires mapping the information in the multivariate distribution of variables such as the stimuli presented to the subject, the neural activity elicited by the presentation of such stimuli, and the behavioral reports of the subject’s perception. More specifically, it requires characterizing the information between the presented stimulus and the behavioral report of the perceived stimulus that can be extracted from neural activity [19]. It is apparent that developing general decompositions of the information exchanged in multivariate systems, as we did here, is key to succeeding in rigorously addressing these fundamental systems-level questions.

Acknowledgments

We are grateful to members of Panzeri’s Laboratory for useful feedback, and to P. E. Latham, A. Brovelli and C. de Mulatier for useful discussions. This research was supported by the Fondation Bertarelli.

Author contributions

All authors conceived the research; G.P., E.P. and D.C. performed the research; G.P., E.P. and D.C. wrote a first draft of the manuscript; all authors edited and approved the final manuscript; S.P. supervised the research.