Bundled Causal History Interaction

TL;DR

This paper introduces an information-theoretic method to analyze how groups of variables in complex systems causally influence each other over time, demonstrated through a chemical system example.

Contribution

It presents a novel approach using partial information decomposition to quantify interactions between variable groups in probabilistic graphical models.

Findings

Successfully applied to a stream chemistry system to reveal complex dependencies.

Quantifies the strength of interactions and memory effects between variable groups.

Establishes a foundation for studying group interactions in various complex systems.

Abstract

Complex system arises as a result of the nonlinear interactions between components. In particular, the evolutionary dynamics of a multivariate system encodes the ways in which different variables interact with each other individually or in groups. One fundamental question that remains unanswered is: how do two non-overlapping multivariate subsets of variables interact to causally determine the outcome of a specific variable? Here we provide an information based approach to address this problem. We delineate the temporal interactions between the bundles in a probabilistic graphical model. The strength of the interactions, captured by partial information decomposition, then exposes complex behavior of dependencies and memory within the system. The proposed approach successfully illustrates complex dependence between cations and anions as determinants of \textit{pH} in an observed stream chemistry system. This example demonstrates the potentially broad applicability of the approach, establishing the foundation to study the interaction between groups of variables in a range of complex systems.

Equations

$${\mathcal{T}}_{mn}=I(X_t^{\text{tar}};{\vec{X}}_{:t}^{mn}\mid {\vec{X}}_{:t}^{\text{rest}}),$$

$${\mathcal{T}}_{mn}=R^{\mathcal{T}}+S^{\mathcal{T}}+U_m^{\mathcal{T}}+U_n^{\mathcal{T}}.$$

$${\mathcal{T}}_{mn}=$$

$$+\underset{={\mathcal{D}}_{mn}}{\underbrace{I(X_t^{\text{tar}};{\vec{X}}_{:t-{\tau }_{\mathbb{C}}}^{mn}\mid {\vec{X}}_{:t}^{\text{rest}})}},$$

$${\mathcal{T}}_{mn}=$$

$${\mathcal{J}}_{mn}=$$

$${\mathcal{D}}_{mn}=$$

$${\mathcal{T}}_{mn}=$$

$${\mathcal{T}}_{mn}$$

$${\mathcal{J}}_{mn}$$

$${\mathcal{D}}_{mn}$$

Full text

I Introduction

In complex systems shaped by the interaction of a multitude of variables, an interesting question that remains unanswered is: In what ways do the evolutionary history of two subsets of variables interactively influence the current state of a target variable? Answering this question would be extremely useful in furthering our understanding in the collective behavior of a system’s dynamics, where the interactions of variables in groups play a key role. For instance, in the study of connectivity between different regions of the brain, one may be interested in how a specific reaction pulse is jointly induced by different groups of incoming signals Tononi and Edelman (1998). In stream chemistry, which is shaped by numerous biophysical processes and chemical reactions in both the stream and the contributing landscape, one may be interested in understanding how stream pH level is an outcome of the joint effect of the concentrations of different anions and cations Kirchner and Neal (2013). Addressing how the state of a specific variable at any time is a causal outcome of interaction from the entire or a part of the evolutionary history of a system requires a quantitative approach.

At the most elementary level, this approach calls upon investigating whether and in what way two variables interact with each other (Figure 1a). This type of pairwise interaction has been widely addressed by causal influence analysis Granger (1969); Pearl (1995); Sugihara et al. (2012); Imbens and Rubin (2015). Generally, most causality detection and analysis fall into two categories: the intervention- and non-intervention-based approaches. While the interventional analysis (e.g., Pearl’s causality Pearl (1995)) is more intuitive by investigating how intervening a source variable impacts the target variable, the non-interventional analysis is more applicable in most real life systems where artificial interventions are almost impossible in a multivariate situation (Figure 1b–d). Among the existing non-interventional approaches, Granger causality Granger (1969) has gained significant popularity due to its intuitive statistical interpretation using observed time series to evaluate the cause–effect relation between two variables. This is done by quantifying the reduced uncertainty of the target that is explained by the source conditioned on the knowledge of the remaining variables in the system. In this study, the causal analysis is referred in this Granger sense.

While the original Granger causality only assesses the change of the second-order moment to determine the reduced uncertainty, recent studies extend it to capture the change of the entire probability space by using information measures Schreiber (2000); Runge et al. (2012). For example, transfer entropy Schreiber (2000) quantifies the shared dependency between the current state of a target and the previous state of a source variable given the knowledge of the previous state of the target. Besides, momentary information transfer Runge et al. (2012) combines multivariate transfer entropy with probabilistic graphical model to efficiently estimate the information flow between two lagged variables in a multivariate system. Information measures, therefore, are the core utilities for quantitative analysis in this study, in that they are capable of delineating the nonlinear interactions.

Here, we propose new information measures to quantify and characterize the interactive strength among two bundled variable sets in affecting the present state of a target variable (Figure 1c). This approach allows us: (1) to consider the effect of the entire evolutionary history of all interacting variables, termed causal history Jiang and Kumar (2018, 2019), that determines the current state of a variable of interest; and (2) to characterize such effect by using partial information decomposition (PID) Williams and Beer (2010) framework. We aim to analyze the joint influence from the evolutionary history of two subsets of variables, thus requiring multivariate interaction assessment. This draws upon another key feature of information measure—characterizing the dynamics among multiple components. For instance, to investigate the total information from two source variables to a target (Figure 1b), PID is used for decomposing the total information into different information contents identified as unique, redundant and synergistic. In addition, we further the PID approach to characterize the information from two lagged sources through specific pathways by conditioning the remaining system’s dynamics that are not in the pathways of interest Jiang and Kumar (2018). This approach is called momentary partial information decomposition, building on the idea of momentary information and PID. Another illustrative example of using information measures to assess multivariate interaction is the causal history analysis framework Jiang and Kumar (2019, ), which accounts for the influence from the entire evolutionary dynamics of the system (Figure 1c).

The main contribution of this study is that it is the first time to delineate the overall effect of multiple interacting grouped sources to a target using information theory. It should be noted that the proposed method is fundamentally different from existing information measures. Information theory has been extensively employed to assess pairwise interactions (e.g., transfer entropy Schreiber (2000) and momentary information transfer Runge et al. (2012)), i.e., whether and how a source affects a target. Some recent efforts take one step further to unravel multivariate interactions (e.g., PID and the previous causal history analysis framework). Nevertheless, neither the pairwise interaction nor the current multivariate analysis using information theory takes into account group interactions, which is the key in this study. By grouping variables, the proposed analysis targets dynamics of specific sets of components, thus providing new insights on interactions among different subsets of a system. Such insights cannot be obtained using previous analysis treating variables as individual components.

The rest of the paper is organized as follows. Section II details the proposed information measures for analyzing the bundled causal history interaction. They are developed based on a directed acyclic graph (DAG) representation for time-series. The DAG further serves as the basis for dimensionality reduction of the measures to ensure reliable information estimations. In Section III, the bundled causal history analysis is employed to investigate the joint influence on the stream pH from cations and anions by using a set of observed stream chemistry data. Last, a brief conclusion is drawn in Section IV.

II Methodology

We consider complex systems that can be conceptualized as a multivariate system consisting of $N$ variables $\vec{X}_{t}=\{X^{(1)}_{t},X^{(2)}_{t},X^{(3)}_{t},\dots,X^{(N)}_{t}\}_{N}$, varying in time $t$. The current state of a target variable, $X^{\text{tar}}_{{t}}\in\vec{X}_{t}$, is an outcome of interactions in the entire evolutionary dynamics in the causal history Jiang and Kumar (2019) prior to time $t$, $\vec{X}_{:t}=\{\dots,\vec{X}_{t-3},\vec{X}_{t-2},\vec{X}_{t-1}\}$. Among the influence from the entire causal history in $\vec{X}_{:t}$, here, we investigate the joint effect from the historical states of two specific groups of variables in $\vec{X}_{:t}$ on $X^{\text{tar}}_{{t}}$. We denote a bundled set containing $N_{m}(<N)$ variables at time $t$ as $\vec{X}^{m}_{t}\subset\vec{X}_{t}$. The entire historical states of $\vec{X}^{m}_{t}$ is represented as $\vec{X}^{m}_{:t}=\{\dots,\vec{X}^{m}_{t-3},\vec{X}^{m}_{t-2},\vec{X}^{m}_{t-1}\}$. Now, the aim of our study is to characterize how $X^{\text{tar}}_{{t}}$ is jointly driven by causal histories of two bundled sets, $\vec{X}^{m}_{:t}$ and $\vec{X}^{n}_{:t}$, where $\vec{X}^{m}_{:t}\cap\vec{X}^{n}_{:t}=\emptyset$.

In the rest of this section, we first develop the information measures for delineating the information flow from the two bundled sets, $\vec{X}^{m}_{:t}$ and $\vec{X}^{n}_{:t}$, to the target $X^{\text{tar}}_{{t}}$. Then, to achieve reliable information measure estimation, we introduce a two-stage dimensionality reduction approach to reduce the cardinality of the proposed measures.

II.1 Interactive Information Flow from Two Bundled Variables

Let us denote the remaining variables in the system outside of the two chosen bundles as $\vec{X}^{\text{rest}}_{t}=\vec{X}_{t}\backslash\{\vec{X}^{m}_{t},\vec{X}^{n}_{t}\}$ with $\backslash$ as the exclusion symbol. The total information, $\mathcal{T}_{mn}$, given by the evolutionary dynamics of two bundled sets to the target $X^{\text{tar}}_{{t}}$ can then be expressed as a conditional mutual information (CMI) Shannon and Weaver (1949) between the two bundled sets given the knowledge of $\vec{X}^{\text{rest}}_{:t}$, which is given by:

[TABLE]

where $\vec{X}^{mn}_{:t}=\vec{X}^{m}_{:t}\cup\vec{X}^{n}_{:t}$. The conditioning on $\vec{X}^{\text{rest}}_{:t}$ is to exclude the influence from the interactions of the rest of the system in the quantification of the interaction between the two bundles. Note that $X^{\text{tar}}_{{t}}$ can belong to any variable in one of the bundled sets or to that in $\vec{X}^{\text{rest}}_{t}$. When $X^{\text{tar}}_{{t}}\in\vec{X}^{\text{rest}}_{t}$, Equation (1) can be considered as a generalized transfer entropy Schreiber (2000). Transfer entropy captures the reduction in the uncertainty associated with the prediction of the current state of a variable given the knowledge of another variable that is in addition to that from the knowledge of its own history. This generalization allows us to characterize the reduction in uncertainty from multiple variables that are in addition to those provided by the variables own history or that of a set to which it belongs.

To characterize the information contents in $\mathcal{T}_{mn}$, we take advantage of Partial Information Decomposition (PID) Williams and Beer (2010), which allows us to decompose $\mathcal{T}_{mn}$ into: (1) redundant information $R^{\mathcal{T}}$—the overlapping information given by two bundled causal histories, $\vec{X}^{m}_{:t}$ and $\vec{X}^{n}_{:t}$; (2) synergistic information $S^{\mathcal{T}}$—the joint information given by $\vec{X}^{m}_{:t}$ and $\vec{X}^{n}_{:t}$; and (3) unique information $U^{\mathcal{T}}_{m}$ and $U^{\mathcal{T}}_{n}$—the information provided by each bundled set, $\vec{X}^{m}_{:t}$ and $\vec{X}^{n}_{:t}$, respectively. This is given by:

[TABLE]

Before we develop quantitative estimation, we ask another related question: Do all the historical states in the bundled sets provide information to $X^{\text{tar}}_{{t}}$? If the answer is no, then when does such influence end as the lag between $X^{\text{tar}}_{{t}}$ and any source node in $\vec{X}^{mn}_{:t}$ increases? Otherwise, how much information is given by very early dynamics in $\vec{X}^{mn}_{:t}$? Answering these questions requires the assessment of the memory dependencies due to the bundled causal history $\vec{X}^{mn}_{:t}$ on $X^{\text{tar}}_{{t}}$. Therefore, we partition the entire bundled causal history into two complementary components: (1) a recent dynamics from all the states up to a positive time lag ${\tau_{\mathbb{C}}}$, $\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}:t}=\{\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}},\dots,\vec{X}^{mn}_{t-1}\}$, termed immediate bundled causal history; and (2) the remaining earlier dynamics, $\vec{X}^{mn}_{:t-{\tau_{\mathbb{C}}}}$, termed distant bundled causal history. By using the chain rule of CMI Cover and Thomas (2006), we can decompose $\mathcal{T}_{mn}$ in Equation (1) into the information from the immediate ($\mathcal{J}_{mn}$) and distant ($\mathcal{D}_{mn}$) bundled causal histories, which are given by:

[TABLE]

Note that information flow from the two partitioned histories, $\mathcal{J}_{mn}$ and $\mathcal{D}_{mn}$ in Equation (3), are functions of ${\tau_{\mathbb{C}}}$. Quantifying $\mathcal{J}_{mn}$ and $\mathcal{D}_{mn}$ along with ${\tau_{\mathbb{C}}}$ allows the investigation of the memory dependency due to the evolutionary interactions of the two bundled set Jiang and Kumar (2019).

Partitioning $\vec{X}^{mn}_{:t}$ into immediate and distant causal histories further highlights the need to characterize the joint interactions of the two bundled sets in the two complementary historical states. This, again, can be achieved by using the PID approach, and is given by:

[TABLE]

where the last equation reflects the sum of the corresponding terms in the previous two equations. Therefore, Equation (4) illustrates the additive contribution of each information content (i.e., synergistic, redundant, and unique components) in the two partitioned histories, $\mathcal{J}_{mn}$ and $\mathcal{D}_{mn}$, to the entire bundled causal history, $\mathcal{T}_{mn}$.

II.2 Two-Stage Dimensionality Reduction

Computing information flows in Equations (1)–(4) is infeasible due to the possibly infinite length of historical states involved, resulting in a joint probability density with infinite dimensions. To resolve this issue, we represent the temporal dependencies of the system as a Directed Acyclic Graph (DAG), where the dimensionality reduction is performed in the following two stages. First, we employ the probabilistic graphical model approach developed by Eichler and Runge Eichler (2012); Runge et al. (2012) that allows a reduction of the infinite historical states in the above equations into a finite set, by assuming Markov property for the DAG. Then, a further dimension reduction is achieved through reducing the DAG by eliminating “redundant” edges. This elimination is performed by using weighted transitive reduction Bosnacki et al. (2010) with momentary information transfer serving as the weights. This approach is called Momentary Information Weighted Transitive Reduction (MIWTR) Jiang and Kumar . The second stage of dimensionality reduction is to avoid potential high albeit finite cardinalities of the joint probability in computing the information measures after the first round of reduction.

Stage 1: From infinite to finite cardinality—a probabilistic graphical model approach. Figure 2 illustrates the use of a DAG for time-series representation. The DAG, $\mathcal{G}=(\vec{X}_{:t},E)$, includes a set of directed edges $E$ and a set of nodes $\vec{X}_{:t}$ connected by edges in $E$. Every state in $\vec{X}_{:t}$ is represented by a node in $\mathcal{G}$, and a directed edge in $E$ connecting from an earlier node $X_{t-\tau_{X}}$ to a recent node $Y_{t-\tau_{Y}}$ ($0\leq\tau_{Y}<\tau_{X}$), $X_{t-\tau_{X}}\rightarrow Y_{t-\tau_{Y}}$, refers to the direct influence from $X_{t-\tau_{X}}$ to $Y_{t-\tau_{Y}}$. An illustration of using the DAG for time-series to depict the temporal multivariate dynamics is shown in Figure 2a through a system consisting of seven components, $\{X^{(m_{1})}_{t},X^{(m_{2})}_{t},X^{(n_{1})}_{t},X^{(n_{2})}_{t},X^{(r_{1})}_{t},X^{(r_{2})}_{t},X^{(r_{3})}_{t}\}$. We consider $X^{(r_{1})}_{t}$ as the target, $\vec{X}^{m}_{:t}=\{X^{(m_{1})}_{:t},X^{(m_{2})}_{:t}\}$ as the first bundled causal history, and $\vec{X}^{n}_{:t}=\{X^{(n_{1})}_{:t},X^{(n_{2})}_{:t}\}$ as the second bundled causal history. The nodes involved in the bundled causal histories are highlighted in blue for immediate bundled causal history $\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}:t}$ up to the partitioning time lag ${\tau_{\mathbb{C}}}$, and in orange for the remaining distant bundled causal history $\vec{X}^{mn}_{:t-{\tau_{\mathbb{C}}}}$. The historical states outsize of those two bundled sets in the system, $\vec{X}^{\text{rest}}_{:t}=\{X^{(r_{1})}_{t},X^{(r_{2})}_{t},X^{(r_{3})}_{t}\}$, are denoted as gray nodes.

Estimation of the quantities in Equations (1) and (3), and therefore the corresponding Equations (2) and (4), are challenging because the condition set has a large dimension due to its potentially very long history. Therefore, to avoid the curse of dimensionality in computing Equations (1)–(4), the Markov property for the DAG for time-series, as developed by Lauritzen et al. Lauritzen et al. (1990), is assumed. Loosely speaking, the Markov property for the graphical model states that a node $X_{t}$ is independent of its non-descendants in $\mathcal{G}$ given the knowledge of its parents, denoted by $P_{X_{t}}=\{Y_{t-\tau}:Y_{t}\in\vec{X}_{t},\tau>0,Y_{t-\tau}\rightarrow X_{t}\}$. By using the Markov property, the information flow from the entire ($\mathcal{T}_{mn}$), the immediate ($\mathcal{J}_{mn}$), and the distant ($\mathcal{D}_{mn}$) bundled causal histories in Equations (1) and (3) can be revised as (see Figure 2b):

[TABLE]

where $\vec{F}=P_{\vec{X}^{mn}_{:t}\cup X^{\text{tar}}_{{t}}}$ is the parent set of all the nodes in the bundled causal history $\vec{X}^{mn}_{:t}$ and the target $X^{\text{tar}}_{{t}}$; $\vec{V}=P_{X^{\text{tar}}_{{t}}}\cap\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}:t}$ is the intersection of the parents of the target, $P_{X^{\text{tar}}_{{t}}}$, and the immediate bundled causal history, $\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}:t}$; and $\vec{W}_{\tau_{\mathbb{C}}}=P_{\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}:t}}\cap\vec{X}^{mn}_{:t-{\tau_{\mathbb{C}}}}$ is the parent set of the immediate history belonging to the distant history. Figure 2b illustrates $\vec{V}$, $\vec{W}_{\tau_{\mathbb{C}}}$, and $\vec{F}$ in blue, orange, and gray nodes, respectively, in the seven-component system. Equation (5) states that while information from immediate and distant bundled causal histories is aggregated at $\vec{V}$ and $\vec{W}_{\tau_{\mathbb{C}}}$ influencing $X^{\text{tar}}_{{t}}$, respectively, the conditioning on $\vec{F}$ blocks the information from the remaining dynamics in the system, $\vec{X}^{\text{rest}}_{:t}$, on the interaction between $X^{\text{tar}}_{{t}}$ and $\vec{X}^{mn}_{:t}$.

The usage of the Markov property successfully reduces the infinite nodes in immediate ($\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}:t}$) and distant ($\vec{X}^{mn}_{:t-{\tau_{\mathbb{C}}}}$) bundled causal histories into two finite sets, $\vec{V}$ and $\vec{W}_{\tau_{\mathbb{C}}}$ in Equation (5), respectively. However, the condition set $\vec{F}$ in Equation (5) still contains possibly infinite nodes, making the computation infeasible. Therefore, we now further adopt two orders of approximations on $\vec{F}$, and explore the corresponding implications. At the zeroth order (Order-0), we assume the condition set to be empty, i.e., $\vec{F}_{0}=\emptyset$. This approach of not conditioning on the states in the remaining variables $\vec{X}^{\text{rest}}_{:t}$ allows the information from $\vec{X}^{\text{rest}}_{:t}$ to influence the estimation of the interaction between the target $X^{\text{tar}}_{{t}}$ and the bundled causal history. At the first order (Order-1) approximation, the condition set is allowed to include the parents of the target in the remaining variables $\vec{X}^{\text{rest}}_{:t}$, i.e., $\vec{F}_{1}=P_{X^{\text{tar}}_{{t}}}\cap\vec{X}^{\text{rest}}_{:t}$, denoted as gray hatched nodes in Figure 2b. The Order-0 approximation mimics the idea of mutual information, which aims at capturing the shared dependency between $X^{\text{tar}}_{{t}}$ and $\vec{X}^{mn}_{:t}$. On the other hand, the Order-1 approximation is consistent with the insight of transfer entropy Schreiber (2000), such that we prevent the influence of the states in the remaining system directly affecting the target, represented by $\vec{F}_{1}$, from characterizing the information flowing from the bundled causal history. Note that the simplification due to the Markov property in Equation (5) and the two approximations in the condition set $\vec{F}_{0}$ and $\vec{F}_{1}$ can be also used in computing the synergistic, redundant, and unique information in Equation (4).

Stage 2: From high to low cardinality—MIWTR approach. The cardinality of Equation (5) can be potentially high in a strongly interacting multivariate system, leading to higher uncertainty in the estimation of information measures. Here, we adopt a recently-proposed Momentary Information Weighted Transitive Reduction approach Jiang and Kumar to further reduce the dimensionality of Equation (5) by simplifying the DAG. The basic idea of MIWTR is to first exclude any “redundant” edges connecting a node in $\vec{W}_{\tau_{\mathbb{C}}}$ with node in immediate history $\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}:t}$ by using weighted transitive reduction, and then remove any node in $\vec{W}_{\tau_{\mathbb{C}}}$ which are now not directly linked to the nodes in $\vec{X}^{mn}_{t-{\tau_{\mathbb{C}}}:t}$, thereby resulting in reduced cardinality of $\vec{W}_{\tau_{\mathbb{C}}}$. Here, the edge weight, representing the information flowing through the edge, is measured by momentary information transfer Runge et al. (2012) which quantifies the shared dependency between two linked nodes conditioned on their parents. The “redundancy” of a directed edge linking two nodes by using WTR, say $X_{t-\tau_{X}}$ to $Y_{t-\tau_{Y}}$, is assessed according to the existence of an indirect path connecting $X_{t-\tau_{X}}$ and $Y_{t-\tau_{Y}}$ as well as the weights of the edges involved. That is, a directed edge, $X_{t-\tau_{X}}\rightarrow Y_{t-\tau_{Y}}$, is considered “redundant” and thus removed, if and only if there exists a path indirectly linking $X_{t-\tau_{X}}$ and $Y_{t-\tau_{Y}}$ and the minimum weight of all the edges in this indirect pathway is larger than that of $X_{t-\tau_{X}}\rightarrow Y_{t-\tau_{Y}}$. In other words, the existence of an indirect pathway, whose capacity of conveying information from $X_{t-\tau_{X}}$ to $Y_{t-\tau_{Y}}$ is stronger than the direct channel between the two nodes, makes the direct edge $X_{t-\tau_{X}}\rightarrow Y_{t-\tau_{Y}}$ obsolete. More details of MIWTR can be found in Jiang and Kumar .

III Application: Bundled Causal Interaction in Stream Chemistry Dynamics

We used this bundled causal history approach to analyze a set of published stream solute data Kirchner and Neal (2013) to understand how two groups consisting of cations and anions affect pH. The data were recorded every 7-h from March 2007 to January 2009, in the Upper Hafren catchment in the United Kingdom. The catchment, approximately 20 km from the western coast, is mainly covered by grassland over acidic soils. To investigate how different cations and anions jointly determine the pH level of the stream, we considered the cations {Na+, Al3+, Ca2+} as the first bundled set, the anions {Cl-, SO42-} as the second bundled set, and {pH, $\ln Q$ (the logarithm of flow rate)} as the remaining variables. Based on the observed data shown in Figure 3a, we constructed the DAG for time-series by using Tigramite algorithm Runge et al. (2012); Runge (2015); Runge et al. (2015, 2017). Generally, the algorithm first builds up preliminary links between nodes by using mutual information-based independence test, and then removes any spurious links by using CMI-based independence test by conditioning on the parents of the connected two nodes. The resulting DAG is shown in Figure 3b, with the estimation methodology for the graph detailed in Jiang and Kumar (2019).

Based on Equation (5), the current state of the target pH, the parents of the target in the bundled causal history ($\vec{V}$), and the parents of the immediate bundled causal history ($\vec{W}_{\tau_{\mathbb{C}}}$) are denoted in black, blue, and orange colors, respectively, in Figure 3b. The Order-1 approximation of the condition set, $\vec{F}_{1}$, is colored in red (note that $\vec{F}_{0}$ is an empty set). Figure 3b shows that $\vec{W}_{\tau_{\mathbb{C}}}$ consists of 23 nodes which results in high dimensionality of the condition set $\vec{W}_{\tau_{\mathbb{C}}}$. Therefore, we reduce the dimensionality of $\vec{W}_{\tau_{\mathbb{C}}}$ using the MIWTR approach (see the Methodology Section for details). The reduced $\vec{W}_{\tau_{\mathbb{C}}}$ obtained by using MIWTR is shown in Figure 3c, where we see that the number of nodes in $\vec{W}_{\tau_{\mathbb{C}}}$ is reduced from 23 to 11.

We next computed the information flow from the entire ($\mathcal{T}_{mn}$), immediate ($\mathcal{J}_{mn}$), and distant ($\mathcal{D}_{mn}$) bundled causal histories in Equation (5) as well as their synergistic, redundant, and unique components in Equation (4) by using $k$-nearest-neighbor ($k$NN) estimator Kraskov et al. (2004). $k$NN estimator was employed in this study because of its better estimation performance in using short dataset compared with other methods (e.g., binning approach and kernel density estimation Khan et al. (2007); Walters-Williams and Li (2009)). To assess the sensitivity of choosing $k$, we computed the information flow in Equations (4) and (5) with $k=[5,6,7,8,9,10,15]$ for both orders of approximations $\vec{F}_{0}$ and $\vec{F}_{1}$ for $\vec{F}$. Different information contents of the PID framework in Equation (4) was estimated using a rescaled approach for calculating the redundant information proposed in Goodwell and Kumar (2017) and also used in Jiang and Kumar (2018). The results are plotted in Figure 4. The figure shows that the information measures corresponding to different $k$ values generally captures similar patterns for each order of approximations. However, the limited data length (shorter than 2000) Kirchner and Neal (2013) and the data gaps shown in Figure 3a results in shorer usable data lengths as ${\tau_{\mathbb{C}}}$ gets larger Jiang and Kumar (2019), and impedes reliable estimation. The limitation in data leads to the large wiggles and peaks in the estimation for some $k$ values, such as the significant drop of $S^{\mathcal{J}}$ in the Order-0 approximation when $k=5,6,7$, and the spike in the Order-1 approximation when $k=15$. This outcome points to the need for further research to investigate the reliability of information metrics estimation by using $k$NN estimator under different data lengths and $k$ values. Here, we chose $k=10$ and 5 for the Order-0 and Order-1 approximations in $\vec{F}$, respectively, for illustration.

The estimated information components from immediate ($\mathcal{J}_{mn}$) and distant ($\mathcal{D}_{mn}$) bundled causal histories, over partitioning time lag ${\tau_{\mathbb{C}}}$ from 5 to 150, are plotted in Figure 5a,b for the Order-0 and Order-1 approximations, respectively. In both approximations of $\vec{F}$, the information from earlier dynamics $\mathcal{D}_{mn}$ converges to a non-zero value with increasing ${\tau_{\mathbb{C}}}$ (the area above the dotted black line). This illustrates the long-term dependence of pH on the selected cation and anion groups, which consist of both unique information ($U^{\mathcal{D}}_{m}$ and $U^{\mathcal{D}}_{n}$) and synergistic information ($S^{\mathcal{D}}$). Moreover, in comparing Order-0 and Order-1 approximations, while larger values of information contents are expected in Order-0 approximation (since there is no conditioning), the different portions of these information in the two approximations reveals Order-1’s condition effect.. When not conditioning on the dynamics from the remaining variables (i.e., $\ln Q$ and pH), Figure 5a shows that cations provide very dominant unique information ($U^{\mathcal{J}}_{m}$ and $U^{\mathcal{D}}_{m}$) to the current state of pH in both histories. This is because cations consists of three of the overall four nodes in $\vec{V}$ in Figure 3, thus dominating the contributions that affect pH. Nevertheless, there still exists a certain amount of redundant information from recent dynamics, $R^{\mathcal{J}}$, and synergy from both histories, $S^{\mathcal{J}}$ and $S^{\mathcal{D}}$. This captures the overlapping and joint effects due to the dynamics of cation and anion concentrations. On the other hand, in the Order-1 approximation, given the knowledge of the historical states in the remaining variables directly affecting pH, that is, $\vec{F}_{1}=\{\ln Q_{t-1},\text{{pH}}_{t-1},\text{{pH}}_{t-3}\}$, we single out the information from the entire bundled causal history transferred only through $\vec{V}$. Preventing the impact from the remaining system on pH, through conditioning on $\vec{F}_{1}$, reduces the total information $\mathcal{T}_{mn}$ significantly, from $\sim$1.3 nats to $\sim$0.14 nats. In particular, the redundancy in immediate bundled causal history, $R^{\mathcal{J}}$ (which is mainly induced by the dependence of solutes on flow rate Jiang and Kumar (2019)), diminishes. This implies no overlapping influence from cations and anions on pH. Meanwhile, the synergistic effect from recent dynamics, $S^{\mathcal{J}}$, now occupies a much larger proportion of the total information, illustrating the interactive influence on pH due to the chemical interactions between cations and anions. This analysis, which illustrates a quantitative way to characterize the information guiding the current state of the stream pH level transferred from the selected cations and anions in the stream, can be generalized to other multivariate systems.

IV Conclusion

We present an information flow-based framework to characterize the joint influence from the evolutionary dynamics of two groups of variables on the present state of a target variable. Partitioning the total information into synergistic, redundant, and unique components helps delineate different information characteristics due to the two bundled sets. This framework was applied to observed stream chemistry datasets, and successfully showed the joint impacts of cations and anions on stream pH.

The proposed information measures are fundamentally different from the causal detection analysis and other existing information measures (see Figure 1). The causal detection techniques (e.g., transfer entropy Schreiber (2000)) are used for analyzing whether and how a source affects a target, or learning a pairwise interaction pattern. Meanwhile, the proposed bundled interaction analysis, rooted in multivariate analysis, aims to characterize the joint outcome of the evolutionary dynamics of two bundled source sets. This unique feature also allows the proposed measures to be distinct from other information measures (e.g., the previous causal history analysis framework Jiang and Kumar (2019)).

One key issue associated with most multivariate analysis is the curse of dimensionality, which impairs reliable estimations on the corresponding measures. Here, we propose a two-stage approach to reduce the dimensions of the information measures. Based on the DAG for time-series, the Markov property is first assumed to reduce the dimensions from infinite in Equation (3) to finite in Equation (5), and a further reduction is performed by simplifying the DAG by using weighted transitive reduction. Furthermore, while the reduced graph might also affect the computed dynamics, we assume that such impact is small compared with the estimation bias induced by the high-dimensionality. This is especially true when the original graph is highly connected and many edges are removed using the two-stage approach, such as the stream chemistry example. The effectiveness of this approach was verified by the application on stream chemistry dynamics.

The proposed two orders of approximation of the influence from the rest of the system further illustrate such characterization under varying impacts from the remaining variables. Characterizing such information in both immediate and distant bundled causal histories, on the other hand, details the delineation of the influence due to a recent and the complementary earlier dynamics. In the stream chemistry application, the analysis shows that the influences of cations and anions on determining the dynamics of pH in the studied catchment are mainly from their synergistic effect and also from their individual impacts through unique information. Such phenomenon is masked (i.e., the top of Figure 5), when the dynamics of the remaining system (i.e., flow rate and earlier history of pH) are not conditioned in the computation of the information measures in Equation (5) (for the case of Order-0 approximations on $\vec{F}$).

With the increasing availability of observed time-series data, such multivariate analysis framework opens new avenues for understanding the role of different groups of components in controlling the dynamics of a complex system.

Acknowledgements.

Funding support from the following NSF grants are acknowledged: EAR 1331906, ICER 1440315, EAR 1417444, and OAC 1835834. The directed acyclic graph for time-series of the stream chemistry example is estimated by using the Tigramite package Runge et al. (2012); Runge (2015); Runge et al. (2015, 2017). The codes for conducting momentary information weighted transitive reduction in Fig. 3 and calculating the information flows in Fig. 5 are available at: https://github.com/HydroComplexity/CausalHistory.