Causal Structure Learning with Conditional and Unique Information Groups-Decomposition Inequalities

TL;DR

This paper introduces new methods to infer causal relationships from data by using information-theoretic constraints, even when some variables are hidden.

Contribution

The paper expands the framework for groups-decomposition inequalities with weaker assumptions and introduces new constraints using unique information measures.

Findings

New groups-decomposition inequalities are derived with weaker independence and configuration requirements.

Constraints with higher inferential power can be derived using hidden variables and data processing inequalities.

An analogous data processing inequality is derived for a measure of conditional unique information.

Abstract

The causal structure of a system imposes constraints on the joint probability distribution of variables that can be generated by the system. Archetypal constraints consist of conditional independencies between variables. However, particularly in the presence of hidden variables, many causal structures are compatible with the same set of independencies inferred from the marginal distributions of observed variables. Additional constraints allow further testing for the compatibility of data with specific causal structures. An existing family of causally informative inequalities compares the information about a set of target variables contained in a collection of variables, with a sum of the information contained in different groups defined as subsets of that collection. While procedures to identify the form of these groups-decomposition inequalities have been previously derived, we substantially enlarge the applicability of the framework. We derive groups-decomposition inequalities subject to weaker independence conditions, with weaker requirements in the configuration of the groups, and additionally allowing for conditioning sets. Furthermore, we show how constraints with higher inferential power may be derived with collections that include hidden variables, and then converted into testable constraints using data processing inequalities. For this purpose, we apply the standard data processing inequality of conditional mutual information and derive an analogous property for a measure of conditional unique information recently introduced to separate redundant, synergistic, and unique contributions to the information that a set of variables has about a target.

Figures

Full text

1. Introduction

The inference of the underlying causal structure of a system using observational data is a fundamental question in many scientific domains. The causal structure of a system imposes constraints on the joint probability distribution of variables generated from it [1,2,3,4], and these constraints can be exploited to learn the causal structure. Causal learning algorithms based on conditional independencies [1,2,5] allow the construction of a partially oriented graph [6] that represents the equivalence class of all causal structures compatible with the set of conditional independencies present in the distribution of the observable variables (the so-called Markov equivalence class). However, without restrictions on the potential existence and structure of an unknown number of hidden variables that could account for the observed dependencies, Markov equivalence classes may encompass many causal structures compatible with the data.

Conditional independencies impose equality constraints on a joint probability distribution; namely, an independence results in the equality between conditional and unconditional probability distributions, or equivalently, in a null mutual information between independent variables. In addition to the information from independencies between the observed variables, causal information can also be obtained from other functional equality constraints [7], such as dormant independencies that would occur under active interventions [8]. Further causal inference power can be obtained incorporating assumptions on the potential form of the causal mechanisms in order to exploit additional independencies associated with hidden substructures within the generative model [9,10], or independencies related to exogenous noise terms [11,12,13]. Other approaches have studied the identifiability of specific parametric families of causal models [3,14]. However, these methods only provide additional inference power if the actual causal mechanisms conform to the required parametric form.

Beyond equality constraints, the causal structure may also impose inequality constraints on the distribution of the data [15,16], which reflect non-verifiable independencies involving hidden variables. Figure 1 illustrates this distinction between pairs of causal structures distinguishable based on independence constraints (Figure 1A,B) and causal structures that may be discriminated based on inequality constraints (Figure 1C,D). The structures of Figure 1A,B belong to different Markov equivalence classes because in Figure 1A variables $[eqn]$ and $[eqn]$ are independent conditioned on S, while in Figure 1B, to obtain an independence it is required to further the condition on $[eqn]$ . On the other hand, the structures of Figure 1C,D belong to the same equivalence class because no independencies exist between the observable variables $[eqn]$ . Nonetheless, if the hidden variables were also observable, these structures would be distinguishable. In Figure 1D, all the dependencies between the observable variables are caused by a single hidden variable U, while in Figure 1C dependencies are created pairwise by different hidden variables. In this case, a testable inequality constraint involving the observable variables reflects the non-verifiable independencies that involve also hidden variables. Intuitively, in Figure 1C, the inequality constraint imposes an upper bound on the overall degree of dependence between the three variables, given that these dependencies arise only in a pairwise manner, while in Figure 1D no such bound exists.

Importantly, unlike equality constraints, inequality constraints provide necessary but not sufficient conditions for the compatibility of data with a certain causal structure. While a certain hypothesized causal structure—like in Figure 1C—may impose the fulfillment of a given inequality intrinsically from its structure, other causal structures—like in Figure 1D—can generate data that, given a particular instantiation of the causal mechanisms, also fulfill the inequality. Accordingly, the causal inference power of inequality constraints lies in the ability to reject hypothesized causal structures that would intrinsically require the fulfillment of an inequality when that inequality is not fulfilled by the data. This means that tighter inequalities have more inferential power, giving the capacity to discard more causal structures.

Two main classes of inequality constraints have been derived. The first class corresponds to inequality constraints in the probability space, which comprise tests of compatibility such as Bell-type inequalities [17,18], instrumental inequalities [19,20], and inequalities that appear on identifiable interventional distributions [21]. The second class corresponds to inequalities involving information-theoretic quantities. The relation between these probabilistic and entropic inequalities has been examined in [22]. One approach to construct entropic inequalities combines the inequalities defining the Shannon entropic cone, i.e., associated with the non-negativity, monotonicity, and submodularity properties of entropy, and additional independence constraints related to the causal structure [23,24]. Additional causally informative inequalities can be derived if considering the so-called Non-Shannon inequalities [25,26]. When the causal structure to be tested involves hidden variables, all non-trivial entropic inequalities in the marginal scenario associated with the set of observable variables can be derived with an algorithmic procedure [23,24] that projects the set of inequalities of all variables into inequalities that only involve the subset of observable variables.

As an alternative approach, information-theoretic inequality constraints can be derived by an explicit analytical formulation [24,27]. In particular, [27] introduced inequalities comparing the information about a target variable contained in a whole collection of variables with a weighted sum of the information contained in groups of variables corresponding to subsets of the collection. Two procedures were introduced to select the composition of these groups. In a first type of inequalities, the composition of the groups is arbitrarily determined, but an inequality only exists under some conditions of independence between the chosen variables, whose fulfillment reflects the underlying causal structure. In a second type, no conditions are required for the existence of an inequality, but the groups must be ancestral sets; that is, must contain all other variables that have a causal effect on any given element of the group. In both cases, [27] showed that the coefficients in the weighted sum of the information contained in groups of variables are determined by the number of intersections between the groups.

In this work, we build upon the results of [27] and generalize their framework of groups-decomposition inequalities in several ways. First, we generalize both types of inequalities to the conditional case, when the inequalities involve conditional mutual information measures instead of unconditional ones. While this extension is trivial for the first type of inequalities, we show that for the second type it requires a definition of augmented ancestral sets. Second, we formulate more flexible conditions of independence for which the first type of inequalities exists. Third, we add flexibility to the construction of the ancestral sets that appear in the second type of inequalities. We show that, given a causal graph and a conditioning set of variables used for the conditional mutual information measures, alternative inequalities exist when determining ancestors in subgraphs that eliminate causal connections from different subsets of the conditioning variables. Furthermore, we determine conditions in which an inequality also holds when removing subsets of ancestors from the whole set of variables, hence relaxing for the second type of inequalities the requirement that the groups correspond to ancestral sets.

Apart from these generalizations, we expand the power of the approach of [27] by considering inequalities whose existence is determined by the partition into groups of a collection of variables that also contains hidden variables. That is, hidden variables can appear not only as hidden common ancestors of the collection but also as part (or even all) of the variables in the collection for which the inequality is defined. To render operational the use of inequalities derived from collections containing hidden variables, we develop procedures that allow mapping those inequalities into testable inequalities that only involve observable variables. While this mapping can be carried out by simply applying the monotonicity of mutual information to remove hidden variables from the groups, this does not work when all variables in the collection are hidden. We show that data processing inequalities [28] can be applied to obtain testable inequalities also in this case, or applied to obtain tighter inequalities than those obtained by simply removing the hidden variables. We illustrate how testable inequalities whose coefficients in the weighted sum depend on intersections among subsets of hidden variables instead of among subsets of observable variables can result into tighter inequalities with higher inferential power.

In order to derive testable groups-decomposition inequalities, we do not only apply the standard data processing inequality of conditional mutual information [28], but we derive an additional data processing inequality for the so-called unique information measure introduced in [29]. This measure was introduced in the framework of a decomposition of mutual information into redundant, unique, and synergistic information components [30]. Recently, alternative decompositions have been proposed to decompose the joint mutual information that a set of predictor variables has about a target variable into redundant, synergistic, and unique components [31,32,33,34,35] (among others). These alternative decompositions generally differ in the quantification of each component and differ in whether the measures fulfill certain properties or axioms. However, in our work, we do not apply the unique information measure of [29] as part of a decomposition of the joint mutual information. Instead, we show that it provides an alternative data processing inequality that holds for different causal configurations than the standard data processing inequality of conditional mutual information. In this way, the unique information data processing inequality increases the capability to eliminate hidden variables in order to obtain testable groups-decomposition inequalities. Accordingly, the groups-decomposition inequalities we derive can contain unique information terms apart from the standard mutual information and entropy measures that appear when considering the constraints of the Shannon entropic cone [23,24].

We envisage the application of the causally informative tests here proposed in the following way. Given a data set, a hypothesized causal structure is selected to test its compatibility with the data. First, the set of inequality constraints enforced by that causal structure is determined. Second, their fulfillment is evaluated from the data and the causal structure is discarded if some inequality does not hold. In the first step, the determination of the set of groups-decomposition inequalities enforced by a causal structure requires at different levels the verification of conditional independencies. This is the case, for example, with the conditional independencies that are necessary conditions for the existence of the first type of inequalities introduced by [27]. If all variables involved were observable, this verification could be conducted directly from the data. However, as mentioned above, we here consider groups-decomposition inequalities that may contain hidden variables as part of the collection of variables, which precludes this direct verification. For this reason, we will work under the assumption that statistical independencies can be assessed from the structure of the causal graph, namely with the graphical criterion of separability between nodes in the graph known as d-separation [36]. That is, we will rely on the assumption that graphical separability is a sufficient condition for statistical independence and hence characterize the set of groups-decomposition inequalities enforced by a causal structure without using the data. Data would only be used in the second step, in which the actual fulfillment of the inequalities is evaluated.

This paper is organized as follows. In Section 2, we review previous work relevant for our contributions. In Section 3.1, we formulate the data processing inequality for the unique information measure. In Section 3.2, we generalize the first type of inequalities of [27], formulating for the conditional case more general conditions of independence for which a groups-decomposition inequality exists. We also apply data processing inequalities to derive testable groups-decomposition inequalities when collections include hidden variables. In Section 3.3, we generalize the second type of inequalities of [27] as outlined above. In Section 4, we discuss the connection of this work with other approaches to causal structure learning and point to future continuations and potential applications. The Appendix contains proofs of the results (Appendix A and Appendix B) and a discussion of the relations between conditional independencies and d-separations required so that the inequalities here derived are applicable to test causal structures (Appendix C).

2. Previous Work on Information-Theoretic Measures and Causal Graphs Relevant for Our Derivations

In this section we review properties of information-theoretic measures and concepts of causal graphs relevant for our work. In Section 2.1, we review basic inequalities of the mutual information and in Section 2.2 the definition and relevant properties of the unique information measure of [29]. We then review in Section 2.3 Directed Acyclic Graphs (DAGs) and their relation to conditional independence through the graphical criterion of d-separation [36,37]. Finally, we review the inequalities introduced by [27] to test causal structures from information decompositions involving sums of groups of variables (Section 2.4). We do not aim to more broadly review other types of information-theoretic inequalities [23,24] also used for causal inference. The relation with these other types will be considered in the Discussion.

2.1. Mutual Information Inequalities Associated with Independencies

We present in Lemma 1 two well-known inequalities that will be used in our derivations. This lemma corresponds to Lemma 1 in [27]. For completion, we provide the proof of the lemma.

Lemma 1.  The mutual information fulfills the following inequalities in the presence of the corresponding independencies:

• $[eqn]$ (Conditional mutual information data processing inequality): Let $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ be four sets of variables. If $[eqn]$ , then it follows that $[eqn]$ .*

• $[eqn]$ (Increase through conditioning on independent sets): Let $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ be four sets of variables. If $[eqn]$ , then $[eqn]$ .*

Proof.  $[eqn]$ is proven applying, in two different orders, the chain rule of the mutual information to $[eqn]$ :

$$[eqn]$$

Since $[eqn]$ and the mutual information is non-negative, this implies the inequality. To prove $[eqn]$ , the chain rule is applied in different orders to $[eqn]$ :

$$[eqn]$$

Since $[eqn]$ and the mutual information is non-negative, this implies the inequality. □

2.2. Definition and Properties of the Unique Information

The concept of unique information as part of a decomposition of the joint mutual information $[eqn]$ that a set of predictor variables $[eqn]$ has about a target (possibly multivariate) variable $[eqn]$ was introduced in [30]. In the simplest case of two predictors $[eqn]$ , this framework decomposes the joint mutual information about $[eqn]$ into four terms, namely the redundancy of $[eqn]$ and $[eqn]$ , the unique information of $[eqn]$ with respect to $[eqn]$ , the unique information of $[eqn]$ with respect to $[eqn]$ , and the synergy between $[eqn]$ and $[eqn]$ . The predictors share the redundant component, the synergistic one is only obtained by combining the predictors, and unique components are exclusive to each predictor. Several information measures have been proposed to define this decomposition, aiming to comply with a set of desirable properties which were not all fulfilled by the original proposal [29,31,32,33]. However, in this work we will not study the whole decomposition but specifically apply the bivariate measure of unique information introduced in [29]. In Section 3.1, we derive a data processing inequality for this measure and in Section 3.2 we show how it can help to obtain testable groups-decomposition inequalities for causal structures for which the standard data processing inequality of the mutual information would not allow elimination of the hidden variables. In this Section, we review the definition of the unique information measure of [29], we provide a straightforward generalization to a conditional unique information measure, and state a monotonicity property that will be used to derive the data processing inequality of the unique information. The unique information of $[eqn]$ with respect to $[eqn]$ about $[eqn]$ was defined as

$$[eqn]$$

where $[eqn]$ is defined as the set of distributions on $[eqn]$ that preserve the marginals $[eqn]$ and $[eqn]$ of the original distribution $[eqn]$ . The notation $[eqn]$ is used to indicate that the mutual information is calculated on the probability distribution Q. We use $[eqn]$ to refer to the unique information of $[eqn]$ with respect to $[eqn]$ , compared to $[eqn]$ , which is the standard conditional information of $[eqn]$ given $[eqn]$ . We use the notation $[eqn]$ instead of the notation $[eqn]$ introduced by [29] to differentiate it from the set notation $[eqn]$ , which indicates the subset of variables in $[eqn]$ that is not contained in $[eqn]$ , since we will also be using this set notation. This unique information measure is a maximum entropy measure, since all distributions within $[eqn]$ preserve the conditional entropy $[eqn]$ , and hence the minimization is equivalent to a maximization of the conditional entropy $[eqn]$ . The rationale that supports this definition is that the unique information of $[eqn]$ with respect to $[eqn]$ about $[eqn]$ has to be determined by the marginal probabilities $[eqn]$ and $[eqn]$ , and cannot depend on any additional structure in the joint distribution that contributes to the dependence between $[eqn]$ and $[eqn]$ [29]. This additional contribution is removed by minimizing within $[eqn]$ .

In a straightforward generalization, we define the conditional unique information given another set of variables $[eqn]$ as

$$[eqn]$$

where $[eqn]$ is the set of distributions on $[eqn]$ that preserve the marginals $[eqn]$ and $[eqn]$ of the original $[eqn]$ . By construction [29], the conditional unique information is bounded as

$$[eqn]$$

This is consistent with the intuition of the decomposition that the unique information is a component exclusive of $[eqn]$ . In Lemma 2, we present a type of monotonicity fulfilled by the conditional unique information. This result is a straightforward extension to the conditional case of the one stated in Lemma 3 of [38]. We include the full proof because it will be useful in the Results section to prove a related data processing inequality for the unique information. To better suit our subsequent use of notation, we consider the two predictors to be $[eqn]$ and $[eqn]$ , and the conditioning set to be $[eqn]$ .

Lemma 2.  The maximum entropy conditional unique information is monotonic on its second argument, corresponding to the non-conditioning predictor, as follows:

$$[eqn]$$

Proof. Consider the distribution $[eqn]$ and its marginal $[eqn]$ . Consider any distribution $[eqn]$ and its marginal $[eqn]$ on $[eqn]$ . Then $[eqn]$ . By monotonicity of the mutual information, $[eqn]$ is lower than or equal to $[eqn]$ . Since $[eqn]$ does not have $[eqn]$ as an argument, it is equal to the information calculated on its marginal $[eqn]$ . Since this holds for any distribution in $[eqn]$ , it holds in particular for the distribution $[eqn]$ that minimizes $[eqn]$ in $[eqn]$ . Since $[eqn]$ belongs to $[eqn]$ , the minimum of $[eqn]$ in $[eqn]$ is equal to or smaller than $[eqn]$ and hence equal to or smaller than $[eqn]$ . □

2.3. Causal Graphs and Conditional Independencies

We here review basic notions of Directed Acyclic Graphs (DAGs) and the relation between causal structures and dependencies. Consider a set of random variables $[eqn]$ . A DAG $[eqn]$ consists of nodes $[eqn]$ and edges $[eqn]$ between the nodes. The graph contains $[eqn]$ for each $[eqn]$ . We refer to V as both a variable and its corresponding node. Causal influences can be represented in acyclic graphs given that causal mechanisms are not instantaneous and causal loops can be spanned using separate time-indexed variables. A path in G is a sequence of (at least two) distinct nodes $[eqn]$ such that there is an edge between $[eqn]$ and $[eqn]$ for all $[eqn]$ . If all edges are directed as $[eqn]$ the path is a causal or directed path. A node $[eqn]$ is a collider in a path if it has incoming arrows $[eqn]$ and is a noncollider otherwise. A node $[eqn]$ is called a parent of $[eqn]$ if there is an arrow $[eqn]$ . The set of parents is denoted $[eqn]$ . A node $[eqn]$ is called an ancestor of $[eqn]$ if there is a directed path from $[eqn]$ to $[eqn]$ . Conversely, in this case $[eqn]$ is a descendant of $[eqn]$ . For convenience, we define the set of ancestors $[eqn]$ as including $[eqn]$ itself, and the set of descendants $[eqn]$ as also containing $[eqn]$ itself.

The link between generative mechanisms and causal graphs relies on the fact that in the graph a variable $[eqn]$ is a parent of another variable $[eqn]$ if and only if it is an argument of an underlying functional equation that captures the mechanisms that generate $[eqn]$ ; that is, an argument of $[eqn]$ , where $[eqn]$ captures additional sources of stochasticity exogenous to the system. If a DAG constitutes an accurate representation of the causal mechanisms, an isomorphic relation exists between the conditional independencies that hold between variables in the system and a graphical criterion of separability between the nodes, called d-separation [36]. Two nodes X and Y are d-separated given a set of nodes $[eqn]$ if and only if no $[eqn]$ -active paths exist between X and Y. A path is active given the conditioning set $[eqn]$ ( $[eqn]$ -active) if no noncollider in the path belongs to $[eqn]$ and every collider in the path either is in $[eqn]$ or has a descendant in $[eqn]$ . A causal structure G and a generated probability distribution $[eqn]$ are faithful[1,2] to one another when a conditional independence between X and Y given $[eqn]$ —denoted by $[eqn]$ —holds if and only if there is no $[eqn]$ -active path between them; that is, if X and Y are d-separated given $[eqn]$ —denoted by $[eqn]$ . Accordingly, faithfulness is assumed in the algorithms of causal inference [1,2] that examine conditional independencies to characterize the Markov equivalence class of causal structures that share a common set of independencies. A well-known example of a system that is unfaithful to its causal structure is the exclusive-OR (X-OR) logic gate, whose output is independent of the two inputs separately but dependent on them jointly.

In contrast to the algorithms that infer Markov equivalence classes, we will show that the applicability of the groups-decomposition inequalities here studied relies on the assumption that d-separability is a sufficient condition for conditional independence. That is, instead of an if and only if relation between d-separability and conditional independence, as required in the faithfulness assumption, it is enough to assume that d-separability implies conditional independence. As we further discuss in Appendix C, this is a substantially weaker assumption, since usually faithfulness is violated due to the presence of independencies that are incompatible with the causal structure. This is the case, for example, of the X-OR logic gate, for which faithfulness is violated because the inputs are separately independent of the output despite each having an arrow towards the output in the corresponding causal graph. Conversely, the X-OR gate complies with d-separability being a sufficient condition for independence, since in the graph only the input nodes are d-separated and the corresponding input variables of the X-OR gate are independent. Despite only requiring that d-separability implies independence, to simplify the presentation of our results in the main text we will assume faithfulness and indistinctively use $[eqn]$ to indicate statistical independence and graphical separability, instead of distinguishing between $[eqn]$ and $[eqn]$ . In Appendix C, we will more closely examine how in the proofs of our results the sufficient condition of d-separability for conditional independencies is enough. An important implication of independencies following from d-separability is that, if variables $[eqn]$ are separately independent from Y—namely $[eqn]$ and $[eqn]$ —because of the lack of any connection between node Y and both nodes $[eqn]$ and $[eqn]$ , then $[eqn]$ cannot be jointly dependent on Y, namely $[eqn]$

$[eqn]$ cannot occur. This is because d-separability between node Y and the set of nodes $[eqn]$ is determined by separately considering the lack of active paths between Y and each node $[eqn]$ and $[eqn]$ . Since the set of paths between Y and $[eqn]$ is the union of the paths between Y and both $[eqn]$ and $[eqn]$ , considering $[eqn]$ jointly does not add new paths that could create a dependence of Y with $[eqn]$ . A dependence can only be created by conditioning on some other variable, which could activate additional paths by activating a collider.

2.4. Inequalities for Sums of Information Terms from Groups of Variables

We now review two results in [27] that are at the foundation of our results. The first corresponds to their Proposition 1. We provide a slightly more general formulation that is useful for subsequent extensions.

Proposition 1.  (Decomposition of information from groups with conditionally independent non-shared components): Consider a collection of groups $[eqn]$ , where each group $[eqn]$ consists of a subset of observable variables $[eqn]$ , being $[eqn]$ the set of all observable variables. For every $[eqn]$ , define $[eqn]$ as the maximal value such that $[eqn]$ has a non-empty intersection where it intersects jointly with $[eqn]$ other distinct groups out of $[eqn]$ . Consider a conditioning set $[eqn]$ and target variables $[eqn]$ . If each group is conditionally independent given $[eqn]$ from the non-shared variables in each other group (i.e., $[eqn]$ ), then the conditional information that $[eqn]$ has about the target variables $[eqn]$ given $[eqn]$ is bounded from below by

$$[eqn]$$

Proof. The proof is presented in Appendix A. It is a generalization to the conditional case of the proof of Proposition 1 in [27] and a slight generalization that allows for dependencies to exist between variables shared by two groups as long as dependencies with non-shared variables do not exist. □

An illustration of Proposition 1 for the unconditional case is presented in Figure 3 of [27], together with further discussion. In Section 3.2 we will provide further illustrations for the extensions of Proposition 1 that we introduce. We will use $[eqn]$ to indicate the maximal values for all groups. We will add a subindex $[eqn]$ to specify the collection if different collections are compared. A trivial refinement of Proposition 1 would consider $[eqn]$ and for each group $[eqn]$ . This may lead to a tighter lower bound by decreasing some values in $[eqn]$ if some intersections between groups occur in $[eqn]$ . We do not present this refinement in order to simplify the presentation.

The second result from [27] that we will be relying on is their Theorem 1. We present a version that is slightly reduced and modified, which is more convenient in order to relate to our own results.

Theorem 1.  (Decomposition of information in ancestral groups.) Let G be a DAG model that includes nodes corresponding to the variables in a collection of groups $[eqn]$ , which is a subset all observable variables $[eqn]$ . Let $[eqn]$ be the collection of ancestors of the groups, as determined by G. For every ancestral set of a group, $[eqn]$ , let $[eqn]$ be maximal, such that there is a non-empty joint intersection of $[eqn]$ and other $[eqn]$ distinct ancestral sets out of $[eqn]$ . Let $[eqn]$ be a set of target variables. Then the information of $[eqn]$ about $[eqn]$ is bounded as

$$[eqn]$$

Proof. The original proof can be found in [27]. □

In contrast to Proposition 1, a generalization to the conditional mutual information is not trivial and will be developed in Section 3.3. We will also propose additional generalizations regarding which graph to use to construct the ancestral sets and conditions to exclude some ancestors from the groups. In their work, [27] conceptualized $[eqn]$ as corresponding to leaf nodes in the graph, for example providing some noisy measurement of $[eqn]$ , with $[eqn]$ being the case of perfect measurement. While this conceptualization guided their presentation, their results were general, and here we will not assume any concrete causal relation between $[eqn]$ and $[eqn]$ . We have slightly modified the presentation of Theorem 1 from [27] to add the upper bound and to remove some additional subcases with extra assumptions presented in their work. The upper bound is the standard upper bound of mutual information by entropy [28]. In the Results, we will also be interested in cases in which $[eqn]$ contains hidden variables, so that $[eqn]$ cannot be estimated. Given the monotonicity of mutual information, the terms from each ancestral group can be lower bounded by the information in the observable variables within each group and $[eqn]$ is used as a testable upper bound.

There are two main differences between Proposition 1 and Theorem 1. First, Theorem 1 does not impose conditions of independence for the inequality to hold. Second, while the value $[eqn]$ of each group $[eqn]$ is determined in Proposition 1 by the overlap between groups, with no influence of the causal structure relating the variables, on the other hand in Theorem 1 the value $[eqn]$ depends on the causal structure, since it is determined from the intersections between ancestral sets. Despite these differences, given the relation between causal structure and independencies reviewed in Section 2.3, both types of inequalities can have causal inference power to test the compatibility of certain causal structures with data.

3. Results

In Section 3.1, we introduce a data processing inequality for the conditional unique information measure of [29]. In Section 3.2, we develop new information inequalities involving groups of variables and examine how data processing inequalities can help to derive testable inequalities in the presence of hidden variables. In Section 3.3, we develop new information inequalities involving ancestral sets. The application of these inequalities for causal structure learning is discussed. As justified in the proofs of our results (Appendix A and Appendix B) and further discussed in Appendix C, our derivations of groups-decomposition inequalities only rely on the assumption that d-separability implies conditional independence. No further assumptions are used in our work, in particular, our application of the unique information measures of [29] does not require any assumption regarding the precise distribution of the joint mutual information among redundancy, unique, and synergistic components.

3.1. Data Processing Inequality for Conditional Unique Information

Proposition 2.  (Conditional unique information data processing inequality): Let $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ be five sets of variables. If $[eqn]$ , then $[eqn]$ .

Proof. Let $[eqn]$ be the original distribution of the variables and define $[eqn]$ as the set of distributions on $[eqn]$ that preserve the two marginals $[eqn]$ and $[eqn]$ . Let $[eqn]$ be the marginal of $[eqn]$ and $[eqn]$ be the set of distributions that preserve the marginals $[eqn]$ and $[eqn]$ . By the definition of unique information (Equation (2))

$$[eqn]$$

Equality $[eqn]$ follows from the chain rule of mutual information. Equality $[eqn]$ holds because $[eqn]$ does not depend on $[eqn]$ and can be calculated with the marginal $[eqn]$ , marginalizing $[eqn]$ on $[eqn]$ . Note that $[eqn]$ . Since $[eqn]$ is null, $[eqn]$ factorizes as $[eqn]$ . For any distribution $[eqn]$ , which preserves $[eqn]$ and $[eqn]$ , a distribution can be constructed as $[eqn]$ , such that $[eqn]$ , since $[eqn]$ continues to preserve $[eqn]$ and $[eqn]$ is preserved by construction. Also by construction, $[eqn]$ for any $[eqn]$ created from any $[eqn]$ . In particular, this holds for the distribution $[eqn]$ constructed from $[eqn]$ that minimizes $[eqn]$ , which determines $[eqn]$ . The distribution $[eqn]$ minimizes the first term in the r.h.s of Equation (4) and, given the non-negativity of mutual information, it also minimizes the second term, hence providing the minimum in $[eqn]$ . Accordingly, $[eqn]$ $[eqn]$ . The monotonicity of unique information on the non-conditioning predictor (Lemma 2) leads to $[eqn]$ . □

A related data processing inequality has already been previously derived for the unconditional unique information in the case of $[eqn]$ , with $[eqn]$ [39]. Differently, Proposition 2 formulates a data processing inequality for the case $[eqn]$ . When $[eqn]$ , Proposition 2 states a weaker requirement for the existence of an inequality, given the decomposition axiom of the mutual information [27]. As we will now see in Section 3.2, Proposition 2 will allow us to apply the unique information data processing inequality in cases in which $[eqn]$ . In particular, $[eqn]$ allows us to obtain a lower bound when $[eqn]$ contains hidden variables that we want to eliminate in order to have a testable groups-decomposition inequality. In contrast, the application of the standard data processing inequality of the mutual information $[eqn]$ requires $[eqn]$ , and hence the two types of data processing inequalities may be applicable in different cases to eliminate $[eqn]$ . This will be fully appreciated in Propositions 5 and 6. Note that this application of the unique information measure of Equation (2) to eliminate hidden variables is not restrained by the role of the measure in the mutual information decomposition and by considerations about which alternative decompositions optimally quantify the different components [30,35].

3.2. Inequalities Involving Sums of Information Terms from Groups

In this section, we extend Proposition 1 in several ways. Propositions 3–6 present subsequent generalizations, all subsumed by Proposition 6. We present these generalizations progressively to better appreciate the new elements. For these Propositions, examples are displayed in Figure 2 and Figure 3 and explained in text after the enunciation of each Proposition. Which Proposition is illustrated by each example is indicated in the figure caption and in the main text. The objective of these generalizations is twofold: First, to derive new testable inequalities for causal structures not producing a testable inequality from Proposition 1. Second, to find inequalities with higher inferential power, even when some already exist. These objectives are achieved introducing inequalities with less constringent requirements of conditional independence and using data processing inequalities to substitute certain variables from $[eqn]$ , so that the conditions of independence are fulfilled or the number of intersections is reduced and lower values in $[eqn]$ are obtained. The first extension relaxes the conditions $[eqn]$ $[eqn]$ required in Proposition 1:

Proposition 3.  (Weaker conditions of independence through group augmentation for a decomposition of information from groups with conditionally independent non-shared components): Consider a collection of groups $[eqn]$ , a conditioning set $[eqn]$ , and target variables $[eqn]$ as in Proposition 1. Consider that for each group $[eqn]$ a group $[eqn]$ exists, such that $[eqn]$ and $[eqn]$ can be partitioned in two disjoint subsets $[eqn]$ such that $[eqn]$ fulfills the conditions of independence $[eqn]$ and $[eqn]$ the conditions $[eqn]$ $[eqn]$ , and such that $[eqn]$ and $[eqn]$ are disjoint. Define the maximal values $[eqn]$ like in Proposition 1 but for the augmented groups $[eqn]$ . Then, the conditional information that $[eqn]$ has about the target variables $[eqn]$ given $[eqn]$ is bounded from below by:

$$[eqn]$$

Proof. The proof is provided in Appendix A. □

The contribution of Proposition 3 is to relax the conditional independence requirements $[eqn]$ . Analogous conditions remain for $[eqn]$ , but $[eqn]$ needs to fulfill the conditions $[eqn]$ $[eqn]$ . This means that the variables in $[eqn]$ are used to separate the variables in $[eqn]$ from other groups. If $[eqn]$ is empty for all i, Proposition 3 reduces to Proposition 1.

Another difference between Propositions 1 and 3 regards the role of hidden variables. Assume that each $[eqn]$ is formed by $[eqn]$ , where $[eqn]$ are hidden variables and $[eqn]$ observable variables. In Proposition 1, the requirement that the variables are observable is not fundamental and could be removed. However, to obtain a testable inequality, monotonicity of mutual information would need to be applied to reduce each term $[eqn]$ to its estimable lower bound $[eqn]$ that does not contain the hidden variables $[eqn]$ . On the other hand, the fulfillment of $[eqn]$ implies $[eqn]$ , and reducing $[eqn]$ to $[eqn]$ can only decrease the number of intersections, and hence $[eqn]$ values are equal or smaller than $[eqn]$ . Therefore, with Proposition 1, there is no advantage in including hidden variables. When testing Proposition 1 for a hypothesis of the underlying causal structure (and related independencies), it is equally or more powerful to use $[eqn]$ than $[eqn]$ .

This changes in Proposition 3, since $[eqn]$ appears in the conditioning side of the independencies that constrain $[eqn]$ . If hidden variables within $[eqn]$ are necessary to create the independencies for $[eqn]$ , it is not possible to reduce each group to its subset of observable variables. Note that, for a hypothesized causal structure, whether the independence conditions required by Proposition 3 are fulfilled can be verified without observing the hidden variables by using the d-separation criterion on the causal graph, assuming d-separation implies independence. The actual estimation of mutual information values is only needed when testing an inequality from the data.

If $[eqn]$ includes hidden variables, in general $[eqn]$ cannot be estimated and $[eqn]$ is used as an upper bound. For the r.h.s. of the inequality, a lower bound is obtained by monotonicity of the mutual information, removing the hidden variables. In general, a testable inequality has the form

$$[eqn]$$

with $[eqn]$ being the observable variables within each group. In the case that $[eqn]$ $[eqn]$ , that is, if the hidden variables do not add information, then a testable tighter upper bound is available using $[eqn]$ . Importantly, the values $[eqn]$ are determined using the groups in $[eqn]$ . Since $[eqn]$ , group augmentation comes at the price that $[eqn]$ are equal or higher than $[eqn]$ , but the conditional independence requirements may not be fulfilled without it. Note also that the partition $[eqn]$ is not known a priori, but determined in the process of finding suitable augmented groups that fulfill the conditions.

We examine some examples before further generalizations. Throughout all figures, we will read independencies from the causal structures using d-separation, assuming faithfulness. In Figure 2A, consider groups $[eqn]$ and $[eqn]$ , and $[eqn]$ . Proposition 1 is not applicable due to $[eqn]$ $[eqn]$ . Augmenting the groups to $[eqn]$ , $[eqn]$ , and $[eqn]$ the conditions of Proposition 3 are fulfilled, as can be verified by d-separation. Coefficients are determined by $[eqn]$ due to the intersection of the groups in U. Note that hidden variables are not restricted to be hidden common ancestors, and here U is a mediator between $[eqn]$ and $[eqn]$ . In Figure 2B, consider groups $[eqn]$ , $[eqn]$ , $[eqn]$ , which do not fulfill the conditions of Proposition 1. Augmenting the groups to $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ the conditions are fulfilled. Maximal intersection values are $[eqn]$ . In both examples the upper bound is $[eqn]$ since $[eqn]$ cannot be estimated due to hidden variables.

We also consider scenarios with more groups. Figure 2C represents $[eqn]$ groups organized in pairs, with subindexes $[eqn]$ indicating two particular pairs. The $[eqn]$ groups are defined in pairs, with $[eqn]$ and $[eqn]$ , $[eqn]$ . The causal structure is the same across pairs, but the mechanisms generating the variables beyond the causal structure can possibly differ. Proposition 1 is not fulfilled since $[eqn]$ $[eqn]$ . Groups can be augmented to $[eqn]$ , $[eqn]$ , for $[eqn]$ . Proposition 3 then holds with $[eqn]$ for all $[eqn]$ groups. The pairs of groups contribute to the sum as $[eqn]$ , which in the testable inequality of the form of Equation (5) reduces to $[eqn]$ . The upper bound to the sum of $[eqn]$ terms is $[eqn]$ . This inequality provides causal inference power because $[eqn]$ for all j is not directly testable. As previously indicated, the inference power of an inequality emanates from the possibility to discard causal structures that do not fulfill it. Note that for this system an alternative is to define N groups instead of $[eqn]$ groups, each as $[eqn]$ . In this case Proposition 1 is already applicable with the coefficients being all 1, since $[eqn]$ for all $[eqn]$ . For this inequality, each of the N groups contributes with $[eqn]$ , and since there are no hidden variables the l.h.s. is $[eqn]$ . However, this latter inequality holds for any causal structure that fulfills $[eqn]$ for all $[eqn]$ . Given that these independencies do not involve hidden variables, they are directly testable from data, so that the latter inequality does not provide additional inference power, in contrast to the former one.

We now continue with further generalizations. Group augmentation in Proposition 3 cannot decrease the values of the maximal number of intersections. We now describe how the data processing inequalities in Lemma 1 $[eqn]$ and Proposition 2 can be used to substitute variables within the groups, potentially reducing the number of intersections. We start with the data processing inequality for the conditional mutual information.

Proposition 4.  (Decomposition of information from groups modified with the conditional mutual information data processing inequality): Consider a collection of groups $[eqn]$ , a conditioning set $[eqn]$ , and target variables $[eqn]$ as in Proposition 1. Consider that for some group $[eqn]$ a group $[eqn]$ exists such that $[eqn]$ , with $[eqn]$ . Define $[eqn]$ as the collection of groups that replaces $[eqn]$ by $[eqn]$ for those following the previous independence condition. If $[eqn]$ fulfills the conditions of Proposition 3, the inequality derived for $[eqn]$ also provides an upper bound for the sum of the information provided by the groups in $[eqn]$ :

$$[eqn]$$

Proof. The proof applies Proposition 3 to $[eqn]$ followed by the data processing inequality of Lemma 1 $[eqn]$ to each term within the sum in which $[eqn]$ and $[eqn]$ are different. Given that $[eqn]$ implies $[eqn]$ , their sum is also smaller or equal. □

Proposition 3 envisaged cases in which the conditions of independence of Proposition 1 were not fulfilled for a collection $[eqn]$ and augmentation allowed fulfilling weaker conditions, even if with higher $[eqn]$ values compared to $[eqn]$ . Proposition 4 is useful not only when the conditions of independence are not fulfilled for $[eqn]$ , but more generally if some values in $[eqn]$ are lower than in $[eqn]$ , hence providing a tighter inequality. Including hidden variables in $[eqn]$ is beneficiary when replacing observed by hidden variables leads to fewer intersections. The procedures of Proposition 3 and 4 can be combined, that is, starting with $[eqn]$ that contains only observable variables, a new collection can be constructed adding new variables and removing others from $[eqn]$ , ending with $[eqn]$ that contains both observable and hidden variables. The collection $[eqn]$ fulfilling the conditions of Proposition 3 may even contain only hidden variables, and a testable inequality is obtained as long as the data processing inequality allows calculating observable lower bounds for all terms in the sum.

Figure 2D–F are examples of Proposition 4. Again we consider cases with N groups with equal causal structure and use indexes $[eqn]$ to represent two concrete groups. In Figure 2D, with $[eqn]$ , Proposition 3 does not apply for $[eqn]$ conditioning on $[eqn]$ because $[eqn]$ $[eqn]$ , for all $[eqn]$ . However, given that $[eqn]$ , each $[eqn]$ can be replaced to build $[eqn]$ , and since $[eqn]$ , for all $[eqn]$ Proposition 3 applies after using Proposition 4 to create $[eqn]$ . A testable inequality is derived with upper bound $[eqn]$ and a sum of terms $[eqn]$ , each being a lower bound of $[eqn]$ given the data processing inequality that follows from $[eqn]$ . The coefficients are $[eqn]$ . Therefore, in this case Proposition 4 results in an inequality when no inequality held for $[eqn]$ . In Figure 2E, the same procedure relies on $[eqn]$ and $[eqn]$ to use $[eqn]$ to create a testable inequality with l.h.s. $[eqn]$ and the sum of terms $[eqn]$ in the r.h.s. with $[eqn]$ . Note that by U, which has no subindex, we represent in Figure 2E a hidden common driver of all N groups, not only the displayed $[eqn]$ . In this example Proposition 3 could have been directly applied without using Proposition 4 if augmenting $[eqn]$ to $[eqn]$ , with $[eqn]$ and $[eqn]$ , since $[eqn]$ . However, $[eqn]$ , since all groups $[eqn]$ intersect in U. Therefore, in this case an inequality already exists without applying Proposition 4, but its use allows replacing $[eqn]$ by $[eqn]$ , hence creating a tighter inequality with higher causal inference power.

In Figure 2F, again we consider $[eqn]$ groups, consisting of N pairs with the same causal structure across pairs and indices $[eqn]$ representing two of these pairs. For groups $[eqn]$ , with $[eqn]$ and $[eqn]$ , Proposition 3 is directly applicable for $[eqn]$ and $[eqn]$ , with $[eqn]$ . The data processing inequalities associated with $[eqn]$ allow applying Proposition 4 to obtain an inequality for the groups $[eqn]$ , which $[eqn]$ .

Proposition 4 relies on the data processing inequality of the conditional mutual information. The data processing inequality of unique information can also be used for the same purpose, and both data processing inequalities can be combined applying them to different groups.

Proposition 5.  (Decomposition of information from groups modified using across different groups the conditional or unique information data processing inequality): Consider a collection of groups $[eqn]$ , a conditioning set $[eqn]$ , and target variables $[eqn]$ as in Proposition 1. Consider a subset of groups such that for $[eqn]$ a group $[eqn]$ exists such that, for some $[eqn]$ , $[eqn]$ , with $[eqn]$ . Define $[eqn]$ as the collection of groups that replaces $[eqn]$ by $[eqn]$ for those following the previous independence conditions. Define $[eqn]$ for the unaltered groups and $[eqn]$ for all groups. If $[eqn]$ fulfills the conditions of Proposition 3, the inequality derived for $[eqn]$ also provides an upper bound for a sum combining conditional and unique information terms for different groups in $[eqn]$ :

$$[eqn]$$

Proof. The proof applies Proposition 3 to $[eqn]$ and then both types of data processing inequalities depending on which one holds for different groups:

$$[eqn]$$

Inequality $[eqn]$ follows from the unique information always being equal to or smaller than the conditional mutual information (Equation (3)). Inequality $[eqn]$ applies the conditional mutual information data processing inequality to those groups with $[eqn]$ different than $[eqn]$ but $[eqn]$ , and the unique information data processing inequality to those groups with $[eqn]$ . □

Proposition 5 is useful when the conditions of independence required to apply Proposition 3 do not hold for $[eqn]$ . It can also be useful to obtain inequalities with higher causal inferential power if $[eqn]$ are smaller than $[eqn]$ , even if Proposition 3 is directly applicable. By definition, the terms $[eqn]$ are equal to or smaller than $[eqn]$ , which can only decrease the lower bound, but the data processing inequality may hold only for the unique information and not the conditional information term. Note that the partition $[eqn]$ can be group-specific and selected such that data processing inequalities can be applied.

Figure 3A shows an example of the application of the data processing inequality of unique information. For $[eqn]$ , Proposition 3 does not apply to $[eqn]$ because $[eqn]$ $[eqn]$ . The data processing inequality of conditional mutual information does not hold with $[eqn]$ $[eqn]$ . This data processing inequality could be used adding to $[eqn]$ the latent common parent in $[eqn]$ , but this variable would be shared by all augmented groups $[eqn]$ , leading to an intersection of all N groups. Alternatively, the data processing inequality holds for the unique information with $[eqn]$ , and $[eqn]$ for all $[eqn]$ . Proposition 5 is applied with $[eqn]$ , $[eqn]$ , and $[eqn]$ , $[eqn]$ . This leads to an inequality with $[eqn]$ as upper bound and the sum of terms $[eqn]$ at the r.h.s. with coefficients determined by $[eqn]$ . In Figure 3B, taking $[eqn]$ $[eqn]$ and defining the conditioning set $[eqn]$ , we have $[eqn]$ $[eqn]$ and $[eqn]$ $[eqn]$ . On the other hand, $[eqn]$ , so that the data processing can be applied with the unique information and Proposition 5 is applied with $[eqn]$ , $[eqn]$ and $[eqn]$ . An inequality exists given that $[eqn]$ , and the testable inequality has an upper bound $[eqn]$ and at the r.h.s. the sum of terms $[eqn]$ , with $[eqn]$ .

In Figure 3C, we examine an example in which groups differ in the causal structure of the conditioning variable $[eqn]$ : For the groups of the type of group i, $[eqn]$ is a common parent of Y and $[eqn]$ . For the groups of the type of k, $[eqn]$ is a collider in a path between Y and $[eqn]$ . Consider M groups of the former type and $[eqn]$ of the latter. We examine the existence of an inequality for groups defined as $[eqn]$ $[eqn]$ , with $[eqn]$ . Proposition 3 cannot be applied to $[eqn]$ because $[eqn]$ $[eqn]$ for all $[eqn]$ . The mutual information data processing inequality is not applicable to substitute $[eqn]$ because $[eqn]$ $[eqn]$ . However, for the M groups like i, the independence $[eqn]$ leads to the data processing inequality $[eqn]$ $[eqn]$ . For these groups, $[eqn]$ and $[eqn]$ . For the $[eqn]$ groups like k, the independence $[eqn]$ leads to $[eqn]$ $[eqn]$ . For these groups $[eqn]$ and $[eqn]$ . In all cases the modified groups are $[eqn]$ , which fulfill the requirement $[eqn]$ for all $[eqn]$ needed to apply Proposition 3. The testable inequality that follows from Proposition 5 has upper bound $[eqn]$ and in the sum at the r.h.s. has M terms of the form $[eqn]$ and $[eqn]$ terms of the form $[eqn]$ . The coefficients are determined by $[eqn]$ .

Proposition 5 combines both types of data processing inequalities, but only across different groups. Our last extension of Proposition 1 combines both types across and within groups. For each group, we introduce a disjoint partition into $[eqn]$ subgroups $[eqn]$ and define $[eqn]$ . Subgroups are analogously defined for $[eqn]$ , also with $[eqn]$ . In general, for any ordered set of vectors we use $[eqn]$ to refer to all elements up to k, where in general $[eqn]$ can be nonempty.

Proposition 6.  (Decomposition of information from groups modified with the conditional or unique information data processing inequality across and within groups): Consider a collection of groups $[eqn]$ , a conditioning set $[eqn]$ , and a target variable $[eqn]$ as in Proposition 1. Consider that for each group $[eqn]$ there are disjoint partitions $[eqn]$ and $[eqn]$ , and a collection of sets of additional variables $[eqn]$ , such that $[eqn]$ for $[eqn]$ . Define the collection $[eqn]$ with the modified groups $[eqn]$ . If $[eqn]$ fulfills the conditions of Proposition 3, the inequality derived for $[eqn]$ also provides an upper bound for sums combining conditional and unique information terms for different groups in $[eqn]$ :

$$[eqn]$$

for $[eqn]$ .

Proof. The proof is provided in Appendix A. □

If $[eqn]$ for all i, then $[eqn]$ , $[eqn]$ , $[eqn]$ , and Proposition 6 reduces to Proposition 3. If $[eqn]$ and $[eqn]$ for all i, we recover Proposition 4, with $[eqn]$ . If $[eqn]$ for all i and $[eqn]$ for some i, we recover Proposition 5, with $[eqn]$ and $[eqn]$ . Like for previous propositions, some groups may be unmodified such that $[eqn]$ .

The tightest inequality results from maximizing across $[eqn]$ each term in the sum. In the proof of Proposition 6 in Appendix A we show that, when increasing $[eqn]$ , the terms $[eqn]$ are monotonically increasing. However, in general $[eqn]$ can contain hidden variables, which means that, to obtain a testable inequality, for each $[eqn]$ each term needs to be substituted by its lower bound that quantifies the information in the subset of observable variables. For each group, the optimal $[eqn]$ leading to the tightest inequality will depend on the subset of observable variables $[eqn]$ and the corresponding values of $[eqn]$ .

Figure 3D shows an example of application of Proposition 6. Like in Figure 3C, there are two types of groups with different causal structure. M groups have the structure of the variables with indexes $[eqn]$ , and $[eqn]$ . The other $[eqn]$ groups have the structure of the variables with indexes $[eqn]$ , and $[eqn]$ . The conditioning set selected is $[eqn]$ . Proposition 3 cannot be applied directly because $[eqn]$ $[eqn]$ for all $[eqn]$ within the M groups, and $[eqn]$ $[eqn]$ for all $[eqn]$ within the $[eqn]$ groups. Proposition 6 applies as follows. For the $[eqn]$ groups, $[eqn]$ with $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ . The independencies $[eqn]$ for $[eqn]$ correspond in this case to $[eqn]$ , for $[eqn]$ . For the other M groups, $[eqn]$ with $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ . The independencies involved are $[eqn]$ , for $[eqn]$ , and $[eqn]$ , for $[eqn]$ .

Proposition 6 applies because with $[eqn]$ defined as $[eqn]$ for the $[eqn]$ groups and $[eqn]$ for the M groups, the requirements of independence of Proposition 3 are fulfilled, in particular $[eqn]$ for all $[eqn]$ . The terms $[eqn]$ for the $[eqn]$ groups are $[eqn]$ and are substituted by lower bounds $[eqn]$ in the testable inequality. For the M groups, we have the subsequent sequence of inequalities: $[eqn]$ ≥ $[eqn]$ ≥ $[eqn]$ ≥ $[eqn]$ . The first inequality follows from the independence for $[eqn]$ , the second from the unique information being equal or smaller than the conditional information, and the third from the independence for $[eqn]$ . Considering that a testable inequality can only contain observable variables, for the M groups the terms in the sum can be $[eqn]$ or $[eqn]$ , depending on which one is higher. The coefficients are determined by $[eqn]$ and the resulting testable inequality has upper bound $[eqn]$ .

Overall, Propositions 4–6 further extend the cases in which groups-decomposition inequalities of the type of Proposition 1 can be derived. Our Proposition 1 extends Proposition 1 of [27] to allow conditioning sets, Proposition 3 further weakens the conditions of independence required in Proposition 1, and Propositions 4–6 use data processing inequalities to obtain testable inequalities from groups-decompositions derived comprising hidden variables, which can be more powerful than inequalities directly derived without comprising hidden variables. In Figure 2 and Figure 3, we have provided examples of causal structures for which these new groups-decompositions inequalities exist. In all these cases, the use of our groups-decomposition inequalities increases the set of available inequality tests that can be used to reject hypothesized causal structures underlying data.

3.3. Inequalities Involving Sums of Information Terms from Ancestral Sets

We now examine inequalities involving ancestral sets as in Theorem 1 of Steudel and Ay [27], which we reviewed in our Theorem 1 (Section 2.4). We extend this theorem allowing for a conditioning set $[eqn]$ and adding flexibility on how ancestral sets are constructed, as well as allowing the selection of reduced ancestral sets that exclude some variables. Like for Theorem 1, we will use $[eqn]$ to indicate the collection of all ancestral sets in graph G from the collection of groups $[eqn]$ .

The extension of Theorem 1 to allow for a conditioning set $[eqn]$ requires an extension of the notion of ancestral set that will be used to determine the coefficients in the inequalities. The intuition for this extension is that conditioning on $[eqn]$ can introduce new dependencies between groups, in particular when a variable $[eqn]$ is a common descendant of several ancestral groups, and hence conditioning on it activates paths in which it is a collider. The coefficients need to take into account that common information contributions across ancestral groups can originate from these new dependencies. At the same time, conditioning can also block paths that created dependencies between the ancestral groups. To also account for this, we will not only consider ancestral sets in the original graph G, but in any graph $[eqn]$ , with $[eqn]$ . The graph $[eqn]$ is constructed by removing from G all the outgoing arrows from nodes in $[eqn]$ . This has an effect equivalent to conditioning on $[eqn]$ with regard to eliminating dependencies enabled by paths through $[eqn]$ in which the variables in $[eqn]$ are noncolliders, since removing those arrows deactivates the paths. To account for these effects of conditioning on $[eqn]$ , for each $[eqn]$ we define an augmented ancestral set of the groups $[eqn]$ as follows:

$$[eqn]$$

We then define the set $[eqn]$ , that is, the set of groups that have some ancestor not independent from some ancestor of $[eqn]$ that is also ancestor of $[eqn]$ , given $[eqn]$ .

For each $[eqn]$ , let $[eqn]$ be the maximal number such that a non-empty intersection exists between $[eqn]$ and $[eqn]$ other distinct augmented ancestral sets of $[eqn]$ . Furthermore, we define $[eqn]$ as the maximum for all $[eqn]$ :

$$[eqn]$$

We will use $[eqn]$ to refer to the whole set of maximal values for all groups. If required, we will use $[eqn]$ to specify that the collection is $[eqn]$ .

In Figure 4A–D, we consider examples to understand the rationale of how $[eqn]$ is determined in inequalities with a conditioning $[eqn]$ . In Figure 4A, for groups $[eqn]$ and $[eqn]$ , the augmented ancestral sets on graph G are $[eqn]$ and $[eqn]$ , which intersect on Z and $[eqn]$ for $[eqn]$ . However, Z is a noncollider in the path creating a dependence between $[eqn]$ and $[eqn]$ , and conditioning on Z renders them independent, so that $[eqn]$ overestimates the amount of information the groups may share after conditioning. Alternatively, selecting $[eqn]$ the ancestral sets are $[eqn]$ and $[eqn]$ , which do not intersect and $[eqn]$ for $[eqn]$ when calculated following Equation (7). A priori, we do not know which graph $[eqn]$ , $[eqn]$ , results in a tighter inequality. Here we see that $[eqn]$ leads to an inequality with more causal inference power than G for Figure 4A. In Figure 4B, Z is a collider between $[eqn]$ and $[eqn]$ , so that conditioning on Z creates a dependence between the groups. If the values $[eqn]$ were determined from the standard ancestral sets, in this case $[eqn]$ , for $[eqn]$ , which do not intersect, leading to unit coefficients. However, the augmented ancestral sets following Equation (7) are $[eqn]$ for $[eqn]$ , so that $[eqn]$ . This illustrates that the augmented ancestral sets are necessary to properly determine the coefficients in inequalities with conditioning sets $[eqn]$ , in this case reflecting that $[eqn]$ and $[eqn]$ can have redundant information.

Figure 4C shows a scenario in which conditioning creates dependencies of Y with $[eqn]$ and $[eqn]$ , which were previously independent. The standard ancestral sets $[eqn]$ and $[eqn]$ would not intersect in any $[eqn]$ , with $[eqn]$ and would lead to unit values for $[eqn]$ . On the other hand, the augmented ancestral sets are $[eqn]$ for $[eqn]$ and $[eqn]$ for $[eqn]$ , for all $[eqn]$ , with $[eqn]$ . This results in $[eqn]$ in all cases, which appropriately captures that the two groups can have common information about Y when conditioning on $[eqn]$ . The example of Figure 4D illustrates why each value $[eqn]$ is determined separately (Equation (7)) first, and only after is the maximum calculated (Equation (8)). Four groups are defined as $[eqn]$ for $[eqn]$ . If $[eqn]$ were to be determined directly from Equation (7) but using $[eqn]$ , instead of using separately $[eqn]$ and $[eqn]$ , then for all the ancestral sets the augmented ancestral set would include all variables, since $[eqn]$ is equal to $[eqn]$ . This would lead to $[eqn]$ . However, that determination would overestimate how many groups become dependent when conditioning on $[eqn]$ , since $[eqn]$ creates a dependence between $[eqn]$ and $[eqn]$ and $[eqn]$ between $[eqn]$ and $[eqn]$ , but no dependencies across these pairs are created. The determination of $[eqn]$ from Equations (7) and (8) properly leads to a tighter inequality than the one obtained if considering jointly both conditioning variables.

Equipped with this extended definition of $[eqn]$ , we now present our generalization of Theorem 1:

Theorem 2.  Let G be a DAG model containing nodes corresponding to a set of (possibly hidden) variables $[eqn]$ . Let $[eqn]$ be a set of observable target variables, and $[eqn]$ a conditioning set of observable variables, with $[eqn]$ . Let $[eqn]$ be a collection of (possibly overlapping) groups of (possibly hidden) variables $[eqn]$ . Consider a DAG $[eqn]$ selected as $[eqn]$ with $[eqn]$ , constructed by removing from graph G all the outgoing arrows from nodes in $[eqn]$ . Following Equation (7), define an augmented ancestral set in $[eqn]$ for each group $[eqn]$ for each variable in the conditioning set, $[eqn]$ . Following Equation (8), determine $[eqn]$ for each group, given the intersections of the augmented ancestral sets $[eqn]$ . Select a variable $[eqn]$ and a group of variables $[eqn]$ , possibly $[eqn]$ . Define the reduced ancestral sets $[eqn]$ for each $[eqn]$ , and the reduced collection $[eqn]$ . The information about $[eqn]$ in this reduced collection when conditioning on $[eqn]$ is bounded from below by

$$[eqn]$$

Proof. The proof is provided in Appendix B. □

Theorem 2 provides several extensions of Theorem 1. First, it allows for a conditioning set $[eqn]$ . Second, given a hypothesis of the generative causal graph G underlying the data, Theorem 2 can be applied to any $[eqn]$ with $[eqn]$ , and hence offers a set of inequalities potentially adding causal inference power. As we have discussed in relation to Figure 4A–D, the selection of $[eqn]$ that leads to the tightest inequality in some cases will be determined by the causal structure, but in general it also depends on the exact probability distribution of the variables. Third, Theorem 2 allows excluding some variables $[eqn]$ from the ancestral sets, although imposing constraints in the causal structure of $[eqn]$ . The role of these constraints is clear in the proof at Appendix B. The case of Theorem 1 corresponds to $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Excluding some variables $[eqn]$ can be advantageous. For example, if $[eqn]$ is univariate and it overlaps with some ancestral sets, as it is the case when some groups include descendants of Y, then the upper bound $[eqn]$ is equal to $[eqn]$ and also $[eqn]$ is equal to $[eqn]$ for all ancestral sets that include Y. Excluding $[eqn]$ provides a tighter upper bound $[eqn]$ and may provide more causal inferential power. Another scenario in which a reduced collection can be useful is when excluding $[eqn]$ removes all hidden variables from $[eqn]$ , such that $[eqn]$ is observable, giving $[eqn]$ as a testable upper bound instead of $[eqn]$ . When comparing inequalities with different sets $[eqn]$ , in some cases the form of the causal structure and the specification of which variables are hidden or observable will a priori determine an order of causal inference power among the inequalities. However, like for the comparison across $[eqn]$ with $[eqn]$ , in general the power of the different inequalities depends on the details of the generated probability distributions. Formulating general criteria to rank inequalities with different $[eqn]$ , $[eqn]$ , and $[eqn]$ in terms of their inferential power is beyond the scope of this work.

Note that we have formulated Theorem 2 explicitly allowing for hidden variables. Also, in Theorem 1 (as a subcase of Theorem 2) the restriction of $[eqn]$ being observable variables can be removed. In any case, the inclusion of hidden variables can only increase the causal inference power if combined with data processing inequalities to obtain a testable inequality. Propositions 4–6 indicate how to possibly tighten an inequality derived from Proposition 1 by substituting $[eqn]$ by a new collection $[eqn]$ that, including hidden variables, leads to $[eqn]$ smaller than $[eqn]$ . The same application of data processing inequalities of the unique and conditional mutual information can be used for Theorem 2 to determine a $[eqn]$ with $[eqn]$ smaller than $[eqn]$ . The use of data processing inequalities is necessary because they allow substituting some of the observable variables by hidden variables, instead of only adding hidden variables. When only adding variables, the number of intersections between ancestral groups can only increase, hence not decreasing $[eqn]$ . On top of this, a testable inequality replaces information terms of ancestral groups by their lower bounds given by observable subsets of variables. This means that, adding hidden variables, the testable inequality will contain the same information terms of the observable variables, but possibly smaller coefficients, hence resulting in a looser inequality. This is not the case any more when hidden variables are not added but instead substitute some of observable variables, thanks to data processing inequalities. This substitution may decrease the number of intersections between ancestral groups, and the coefficients in the sum may be higher. We will not describe this procedure in detail, since the use of data processing inequalities is analogous to their use in Propositions 4–6.

We now illustrate the application of Theorem 2. In Figure 4E, with $[eqn]$ , the conditions of independence required by Proposition 6 do not hold for any set of groups, either $[eqn]$ , $[eqn]$ , or, with $[eqn]$ , $[eqn]$ , $[eqn]$ or $[eqn]$ , $[eqn]$ . No data processing inequalities can be applied to replace some variables to fulfill the conditions. On the other hand, Theorem 2 can always be applied, since it does not require the fulfillment of some conditions of independence. For example, for $[eqn]$ , $[eqn]$ and for $[eqn]$ , we have $[eqn]$ , $[eqn]$ , $[eqn]$ , and following Equation (7) $[eqn]$ , $[eqn]$ , and $[eqn]$ , for $[eqn]$ . This leads to $[eqn]$ . For illustration purpose, we focus on $[eqn]$ equal to $[eqn]$ or any of its subsets. In all cases $[eqn]$ , for $[eqn]$ , contributing terms $[eqn]$ and $[eqn]$ . For $[eqn]$ or $[eqn]$ , the contribution of the observable lower bound of the third group is $[eqn]$ . For $[eqn]$ or $[eqn]$ , the third group contributes $[eqn]$ . For $[eqn]$ , $[eqn]$ , which is observable and the upper bound is $[eqn]$ . For any other subset of $[eqn]$ the upper bound in the testable inequality is $[eqn]$ . Because the terms in the sum for groups 1 and 2 are equal for all the $[eqn]$ compared, in this case it can be checked that selecting $[eqn]$ leads to the tightest inequality. This example illustrates the utility of being able to construct inequalities for reduced ancestral sets.

While in the previous example only Theorem 2 and not Proposition 6 was applicable, more generally, a causal structure will involve the fulfillment of a set of inequalities, some obtained using Proposition 6 and some using Theorem 2. Which inequalities have higher inferential power will depend on the causal structure and the exact probability distribution of the data, and neither Theorem 2 nor Proposition 6 are more powerful a priori. In Figure 4F, Proposition 6 cannot be applied using $[eqn]$ , $[eqn]$ and conditioning on Z, because $[eqn]$ $[eqn]$ , $[eqn]$ and no data processing inequalities help to substitute these variables. On the other hand, Theorem 2 can be applied with $[eqn]$ , leading to $[eqn]$ , $[eqn]$ , and $[eqn]$ , for all $[eqn]$ . The augmented ancestral sets are $[eqn]$ $[eqn]$ , and $[eqn]$ , also for all $[eqn]$ , resulting in $[eqn]$ . Focusing on the case of $[eqn]$ , or any subset of it, in all cases the associated testable inequality has $[eqn]$ as upper bound and in the r.h.s. the sum of terms $[eqn]$ , $[eqn]$ . Alternatively, defining $[eqn]$ and $[eqn]$ , Proposition 3 is applicable with the two groups intersecting in $[eqn]$ and $[eqn]$ . The associated testable inequality has the same upper bound $[eqn]$ and in the r.h.s. the sum of terms $[eqn]$ and $[eqn]$ . In this case, which inequality has more causal inferential power will depend on the exact distribution of the data.

Overall, Theorem 2 extends Theorem 1 of [27], allowing conditioning sets and providing more flexible conditions to form the groups. In the examples of Figure 4, we have illustrated how Theorem 2 substantially increases the number of groups-decomposition inequalities that can be tested to reject hypothesized causal structures to be compatible with a certain data set.

4. Discussion

We have presented several generalizations of the type of groups-decomposition inequalities introduced by [27], which compare the information about target variables contained in a collection of variables with a weighted sum of the information contained in subsets of the collection. These generalizations include an extension to allow for conditioning sets and methods to identify existing inequalities that involve collections and subsets selected with less restrictive criteria. This comprises less restrictive conditions of independence, the use of ancestral sets from subgraphs of the causal structure of interest, and the removal of some variables from the ancestral sets. We have also shown how to exploit inequalities identified for collections containing hidden variables—which are not directly testable—by converting them into testable inequalities using data processing inequalities.

Our use of data processing inequalities to derive testable groups-decomposition inequalities when collections contain hidden variables is not entirely new. We found inspiration for this approach in the proof of Theorem 1 in [24]. This theorem derives a causally informative inequality from a particular type of causal structure, namely common ancestor graphs in which all dependencies between observable variables are caused by hidden common ancestors. The inequality presented in the theorem corresponds to the setting of a univariate target variable and groups composed by different single observable variables. In their simplest case, each hidden ancestor only has two children, which are observable variables. Their proof uses the mutual information data processing inequality to convert a sum of information terms involving the observable variables into a sum of terms involving the hidden ancestors. The final inequality can equally be proven applying our Proposition 4 by deriving an inequality for the collection of hidden variables and then converting it into a testable inequality using data processing inequalities. The same final inequality can also be derived as an application of our Theorem 2 followed by the use of the data processing inequality.

We have expanded the applicability of data processing inequalities by showing that this type of inequality also holds for conditional unique information measures [29]. For a given causal structure, a testable causally informative inequality may be obtained substituting hidden variables by observable variables thanks to the data processing inequality of the unique information, in cases in which the data processing inequality of mutual information is not applicable. As shown in Proposition 6, the unique information data processing inequalities are particularly powerful for deriving groups-decomposition inequalities with a conditioning set, since they can iteratively be applied to replace different subsets of hidden variables by observable variables choosing which variables are kept as conditioning variables and which ones are taken as reference variables for different unique information measures. This use of unique information indicates how other types of information-theoretic measures could be similarly incorporated to derive causally informative inequalities. Recent developments in the decomposition of mutual information into redundant, synergistic, and unique contributions [30] provide candidate measures whose utility for this purpose needs to be further explored [31,32,33,34,35,40,41] (among others). Furthermore, while this type of decomposition has been extensively debated recently [35,42,43], aspects of its characterization are still unsolved and an understanding of how the terms are related to the causal structure can provide new insights.

One particular domain in which our generalizations can be useful is to study causal interactions among dynamical processes [23,44], for which causal interactions are characterized from time series both in the temporal [45] and spectral domain [46,47,48]. When studying high-dimensional multivariate dynamical processes, such as brain dynamics (e.g., [49,50,51]) or econometric data [52,53], an important question is to determine whether correlations between time series are related to causal influences or to hidden common influences. For highly interconnected systems with many hidden variables, the number of independencies may be small, hence providing limited information about the causal structure. In this case, inequality constraints can help to substantially narrow down the set of causal structures compatible with the data. Accordingly, our generalization to formulate conditional inequalities may play an important role in combination with measures to quantify partial dependencies between time series [54,55]. We expect this approach to be easily adaptable to non-stationary time-series, as it is often the case in the presence of unit roots and co-integrated time series [56,57,58]. This can be carried out by selecting collections and groups consistent with the temporal partitioning in non-stationary information-theoretic measures of causality in time-series [59,60]. Another area to extend the applicability of our proposal is to study non-classical quantum systems [16,61,62,63]. In this case, an extended d-separation criterion [64] and adapted faithfulness considerations [65] have been proposed to take into account the particularities of quantum systems. Further exploration will be required to determine if and how our derivations that rely on d-separation leading to statistical independence (Appendix C) are also applicable when considering generalized causal structures for quantum systems.

Besides the extension to particular domains, an important question yet to be addressed regards the relation between the causal inferential power of different inequalities. Our proposal considerably enlarges the number of groups-decomposition inequalities of the type of [27] available to test the compatibility of a causal structure with a given data set. We have seen in our analysis some examples of how, under certain conditions, the causal structure imposes an ordering to the power of alternative inequalities. Future work should aim to derive broader criteria to rank the inferential power of inequalities, for example in terms of the relation between the conditioning sets or the constituency of the groups that appear in each inequality. Formulating criteria to rank the inferential power of different inequalities would help to simplify the set of inequalities that needs to be tested when the compatibility of a certain causal structure with the data is to be examined.

Apart from a characterization of how groups-decomposition inequalities are related among themselves, future work should also examine the relation and embedding of this type of inequalities with those derived with other approaches. In our understanding, the algorithmic projection procedure of [23,24] could equally retrieve some of the inequalities here described, but without the advantage of having a constructive procedure to derive the form of an inequality directly reading a causal graph, and instead requiring costly computations that may limit the derivation of inequalities for large systems. The incorporation of constraints for other types of information-theoretic measures, such as constraints involving unique information measures, would require an extension of the algorithmic approach. Among other approaches, the so-called Inflation technique [66] stands out as capable of providing asymptotically sufficient tests of causal compatibility [67]. The inflation method creates a new causal structure with multiple copies of the original structure and symmetry constraints on the ancestral properties of the different copies, in such a way that testable constraints on the inflated graph can be mapped back to the compatibility of the original causal structure. However, despite the ongoing advances in its theoretical developments and implementation [68], to our knowledge it is not straightforward to identify the order of inflation and the specific inflation structure adequate to discriminate between certain causal structures. The availability of inequalities easily derived by reading the original causal structure can also be helpful in combination with the inflation method, in order to discard as many candidate causal structures as possible before the design of additional inflated graphs. The connection with other approaches [69,70,71,72,73,74] also deserves further investigation, ultimately to determine minimal sets of inequality constraints with equivalent inferential power.

Beyond the derivation of existing testable causally informative inequalities, a crucial issue for their application is the implementation of the corresponding tests. This implementation depends on the estimation of information-theoretic measures from data. A ubiquitous challenge for the application of mutual information measures is that they are positively biased and their estimation is data-demanding [75,76]. These biases scale with the dimensionality of the variables, and hence can hinder the applicability of information-theoretic inequalities for large collections of variables, or for variables with high cardinality. However, recent advances in the estimation of mutual information for high-dimensional data can help to attenuate these biases [77]. Furthermore, the implementation of the tests can take advantage of the existence of both upper-bound and lower-bound estimators of mutual information [78], using opposite bounds at the two sides of the inequalities. These technical aspects of the implementation of the tests are important to apply all types of information-theoretic inequalities [23,24,25,26,27,71]. Despite these common challenges, our extension of groups-decomposition inequalities does not come at the price of having to test inequalities that intrinsically are more difficult to estimate. Our contribution can substantially increase the number of inequalities available to be tested, and we have provided examples in Figure 2, Figure 3 and Figure 4 of new inequalities in which—in particular thanks to the use of data processing inequalities—the dimensionality of the collections is not increased. Future work is required to determine how to efficiently combine all available tests. In the goal to determine minimal sets of inequality tests that are maximally informative, the statistical power of the tests will need to be considered together with their discrimination power among causal structures.