Information Flow in Computational Systems
Praveen Venkatesh, Sanghamitra Dutta, Pulkit Grover
TL;DR
This paper introduces a rigorous, information-theoretic framework for defining and detecting the flow of information in computational systems modeled as directed graphs, addressing limitations of previous measures.
Contribution
It proposes a new definition of information flow based on conditional mutual information that guarantees the existence of information paths and provides an algorithm for detection.
Findings
The new definition guarantees unbroken information paths.
The framework clarifies when information flow measures are valid.
An algorithm for identifying information paths in noiseless systems is provided.
Abstract
We develop a theoretical framework for defining and identifying flows of information in computational systems. Here, a computational system is assumed to be a directed graph, with "clocked" nodes that send transmissions to each other along the edges of the graph at discrete points in time. We are interested in a definition that captures the dynamic flow of information about a specific message, and which guarantees an unbroken "information path" between appropriately defined inputs and outputs in the directed graph. Prior measures, including those based on Granger Causality and Directed Information, fail to provide clear assumptions and guarantees about when they correctly reflect information flow about a message. We take a systematic approach---iterating through candidate definitions and counterexamples---to arrive at a definition for information flow that is based on conditional mutual…
| Variable(s) | Meaning |
|---|---|
| The original complete directed graph, prior to time-unrolling | |
| The time-unrolled graph making up the computational system | |
| The set of all time points, | |
| The set of all nodes in the computational system | |
| The subset of nodes at time | |
| A node in the graph at time | |
| A node in the original complete directed graph , or a node in the computational system at an unspecified time point | |
| Some subset of nodes in | |
| The set of all edges in the computational system | |
| The set† of all edges at time | |
| Some subset‡ of edges in | |
| An edge in the computational system at time | |
| An edge in the original complete directed graph , or an edge in the computational system at an unspecified time point | |
| The random variable representing the transmission on the edge | |
| Short-hand notation for (refer Equation (1)) | |
| The set of all incoming edges of | |
| The set of all outgoing edges of | |
| The intrinsically generated random variable at the node | |
| The “message”, a random variable that enters the system at time , and whose information flow we seek to understand (refer Definition 3c) | |
| The input nodes: the subset of nodes at time 0 whose outgoing transmissions depend on the message (refer Definition 3c) | |
| The function computed by the node (refer Definition 3b) | |
| †Script forms typically denote sets | |
| ‡Primed script forms typically denote subsets | |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Information Flow in Computational Systems
Praveen Venkatesh∗, Sanghamitra Dutta† and Pulkit Grover‡
Electrical & Computer Engineering, and the Center for the Neural Basis of Cognition
Carnegie Mellon University
∗[email protected] †[email protected] ‡[email protected]
Abstract
We develop a theoretical framework for defining and identifying flows of information in computational systems. Here, a computational system is assumed to be a directed graph, with “clocked” nodes that send transmissions to each other along the edges of the graph at discrete points in time. We are interested in a definition that captures the dynamic flow of information about a specific message, and which guarantees an unbroken “information path” between appropriately defined inputs and outputs in the directed graph. Prior measures, including those based on Granger Causality and Directed Information, fail to provide clear assumptions and guarantees about when they correctly reflect information flow about a message. We take a systematic approach—iterating through candidate definitions and counterexamples—to arrive at a definition for information flow that is based on conditional mutual information, and which satisfies desirable properties, including the existence of information paths. Finally, we describe how information flow might be detected in a noiseless setting, and provide an algorithm to identify information paths on the time-unrolled graph of a computational system.
1 Introduction
1.1 Motivation
Neuroscientists111A short version of this paper has appeared in the 2019 IEEE International Symposium on Information Theory [1]. often seek an understanding of how information flows in the brain while it performs a particular task [2, 3, 4, 5]. As a concrete example, consider the experiment performed by Almeida et al. [2], where they examine how images of common handheld tools are processed in the brain. In simple terms, the question they investigate is this: when attempting to identify a handheld tool, does one make use of knowledge of how to manipulate it? Two hypotheses present themselves:
(i) the answer to the above question is yes, so we should expect that information about a tool’s identity first flows from visual cortex to motor cortex (the area responsible for processing manipulation), before synthesis of visual and motor information occurs at the area of the brain responsible for object recognition.
(ii) The answer to the aforementioned question is no, so we should expect that the information about tools’ identities first flows from visual cortex to the area responsible for object recognition, after which this information arrives at motor cortex.
Thus, distinguishing between these hypotheses is equivalent to determining the path along which information about a tool’s identity flows in the brain. What methods can neuroscientists use to gain such an understanding? What formal theory underlies such an analysis? How does one mathematically define colloquially-used terms such as “information flow”? These are the fundamental questions we try to answer in this paper.
Information flow is a concept that appears in several contexts, across fields ranging from communication systems, control theory and neuroscience to security, algorithmic transparency, and deep learning. While our primary motivation comes from neuroscience, the theory that we develop is broadly applicable to any system which can be modeled in the form of a directed graph, with nodes that communicate functions of their inputs to other nodes, and where transmissions are observable. For example, several kinds of social networks readily fit this bill, and one might wish to analyze how information spreads in such networks. Our framework is also general enough to analyze information flow in various kinds of Artificial Neural Networks: this could be useful for identifying specific paths that carry information distinguishing two or more classes, or for intelligently pruning an Artificial Neural Network post-training.
In the field of neuroscience, studying normal and diseased brain function involves gaining insight into how information is processed in the brain. Attaining such insight, in turn, may require determining how information flows between various parts of the brain. Thus, a nuanced understanding of information flow in the brain could help with diagnosing and treating brain diseases: a subject that is currently of immense interest with numerous efforts around the world [6, 7, 8, 9]. More generally, an understanding of information flow is essential when considering how one might intervene to affect the output of a computational system, be it modulating how information spreads in a social network, or complementing dysfunctional components of the nervous system through stimulation (such as in retinal and cochlear implants).
1.2 Our Goal and Approach
Our overarching goal in this paper is develop a formal theory for understanding information flow in neuroscientific experiments. In order to properly scope our task, we choose to restrict our attention to “event related experimental paradigms” [10], a set of standard neuroscientific experimental design protocols where a stimulus is shown to an animal subject or human participant, whose brain signals are being recorded. This restriction also allows us to decide precisely what kind of information flow we are interested in, since in general, the phrase “information flow” can refer to more than one notion in neuroscience. We identify two dominant interpretations of “information flow”:
(i) the first refers to information about a specific quantity or variable that is of interest to the experimentalist, which in this paper we refer to as the “message”;
(ii) the second refers to information in the abstract, and is usually used to describe the fact that one area of the brain “drives” or “influences” another area through the transmission of some information: in this interpretation, one is not interested in what is being communicated, only that the communication is occurring.
In this paper, we focus only on the first interpretation of the phrase, where we are interested in information about a specific message, and we wish to track how information about this message flows within the brain. All references to “information flow”, henceforth, refer only to the first interpretation. This is particularly common in event-driven paradigms, where the neuroscientist investigates how the brain responds to a carefully chosen set of stimuli, and examines how information contained in these stimuli (or alternatively, information contained in the response) flows through the brain.
Given that we are interested in information about a specific message, what are we in pursuit of when we say information flow? Broadly speaking, we want to develop a measure that will allow us to examine how and at what times information about a specific message flows from one area of the brain to another. In particular, we think of the brain as a computational system executing an algorithm, and we want to capture how information about different variables might flow between different computational nodes of this system. A given “message” variable may be stored at a particular node for some time, a function may be computed using this message, and then the result may be passed on to a different node. A node that transmits information about the message at one time instant may not do so at a later time instant; thus information flow is a dynamic or time-dependent quantity. The computational system should allow for all of these possibilities, and our ensuing measure of information flow should enable us to track the path traversed by the message222or information derived thereof through this system, over time. This is the principal goal of our theoretical development, and will guide many of our decisions in model design.
We approach this goal by formally defining a computational system model: one based on nodes that represent distinct computational areas of the brain. These nodes can potentially represent the brain at any scale: single neurons, groups of neurons, or even whole brain regions, depending on the measurement modality and the kind of experiment being performed. The computational model we develop borrows various ideas from across several fields. The basic ingredients of the computational model, based on a graph with computational nodes, derives from Thompson’s work on VLSI complexity theory [11]. In order to attain a dynamic picture of information flow on the edges of the computational graph, and to deal with cycles in the flow of information, we use the idea of time-unrolling a graph, taking inspiration from Network Information Theory [12]. Finally, to describe how the computational nodes can compute stochastic functions of variables based on their current inputs, we use the idea of Structural Causal Models from the field of Causality [13, 14].
Within this computational model, we define a new measure for information flow about a specific message that captures the dynamic nature of information transmission. Ultimately, the measure should allow us to track how information about the message flows through the system, in the form of an unbroken information path—this will be a key focus guiding our definitions. Given the nature of the problem, we rely on information-theoretic measures to define information flow. We motivate this definition through properties, and provide a series of candidate definitions and counterexamples before arriving at our final definition. When defining information flow in such a computational system, we restrict ourselves to “observational” measures, which can be computed from a sample of all random variables described in the model. We deliberately eschew interventional and counterfactual measures, as the former require the capability to intervene on the system and change the distributions of the random variables involved, while the latter are a purely theoretical notion that can only be applied in situations where one can ask what might have occurred if a specific variable had been different on a particular trial (while keeping the realizations of all other latent sources of randomness fixed).
The approach of building a rigorous theoretical framework that we have adopted in this paper is inspired by two works from biologists titled “Can a biologist fix a radio?” [15] and “Could a neuroscientist understand a microprocessor?” [16]. Both these works point to the lack of formal methods, i.e., systematic theory, that could help biologists understand the limitations of their tools and test their assumptions. It is our belief that information theory can help provide the formal methods that are sought in biology, and make an impact in fields such as neuroscience and neuroengineering [17, 18, 19]. In particular, information theory can play an important role in advancing how we understand large computational systems through external measurements and interventions. While developing an understanding of information flow in such systems may not be sufficient for providing a complete description of the nature of computation itself, we believe that it forms an integral component. Going forward, we believe that providing a formal theoretical framework for information flow is but a small part of several larger theoretical questions that are yet to be properly posed: questions such as how one might formalize “reverse engineering” the brain, or formalize the notion of “understanding computation”.
1.3 Related Work
Prior work on statistically inferring flows of information in the brain appears under the umbrella of “functional” or “effective connectivity” [20, 21, 22]. These efforts have largely relied on measures of statistical333We borrow the use of the term “statistical” from Pearl [13, Sec. 1.5], who contrasts and differentiates “statistical” concepts from (strictly) “causal” ones. causal influence such as Granger Causality [23, 24], Massey’s Directed Information [25, 26, 27, 28], Transfer Entropy [29] and Partial Directed Coherence [30]. Despite widespread use, these measures have frequently been a subject of debate and disagreement within the neuroscientific community [31, 32, 33, 34, 35, 36, 37]. In part, these disagreements stem from the widely-acknowledged fact that under non-ideal measurement conditions (e.g. in the presence of hidden variables [13, p. 54], asymmetric noise [38, 39], or limited sampling [40]), estimation of these quantities may be erroneous. While these non-idealities may eventually be overcome through improvements in technology, we believe that more fundamental issues still remain. For instance, one basic question that has remained unanswered is: when can statistical causal influence be interpreted as information flow about a message? In previous work, we demonstrated that even under ideal measurement conditions, the direction of greater Granger causal influence can be opposite to the direction in which the message is being communicated in certain kinds of feedback communication networks [41]. This example points to a more general issue with the use of statistical causal influence measures: there is no direct way to interpret what the influence is “about”. While it is understood in certain settings that “information flow” refers to information contained in a particular set of “stimuli” (as mentioned in the previous section), the aforementioned measures do not incorporate the effect of the stimulus.
The existence of such fundamental issues can be traced back to the fact that there is no underlying model that links information flow (of some message of interest) with the signals that are actually measured, leading to a lack of separation between the problems of defining information flow and of estimating it. The lack of such a computational model also makes it hard to test assumptions and to draw the right interpretations from experimental analyses. We believe that, following Shannon’s approach of providing a theoretical foundation for information transmission [42], a solid theoretical treatment of information flow is needed. Such a treatment would begin with a model of the underlying system, give a definition for information flow and describe its properties, and finally end with a suitable estimator. Adopting Shannon’s model of defining entropy by stating a set of properties that such a measure must satisfy, we attempt to define information flow by putting forward an intuitive property that we believe is desirable for such a quantity. It is our hope that, by providing a theoretical foundation that separates definition and estimation, along with a concrete model and explicitly-stated assumptions, we can avoid many of the pitfalls encountered by previous approaches to understanding information flow in the brain.
It is useful at this point to mention the key differences between our measure of information flow, and measures based on Granger Causality and its generalizations:
Our measure depends on a message , that will often be related to the stimulus or the response in a neuroscientific task, whereas tools based on Granger causality do not. 2. 2.
Since Granger causality-based tools use time series modeling to compute an estimate of information flow, they are unable to provide a dynamic, evolving picture of information flow between different areas over time. 3. 3.
Since we start with a computational framework, our model provides a direct way to connect information flow with the underlying computation. On the other hand, Granger causality-based tools start with a probabilistic graphical model of the observed nodes, and do not tie the analysis to computation in any way.
While our proposed definition of information flow will also suffer from performance degradation under non-ideal measurement conditions, we believe that it overcomes the fundamental difficulty faced by Granger Causality-based tools: when measurements are ideal, our definition provides a clear and consistent way to interpret information flow about a message, as we illustrate through several examples in Section 6.
Another line of work that appears within the functional and effective connectivity literature is Dynamic Causal Modeling (DCM) [43, 21]. This methodology is, in spirit, much more closely aligned with what we propose here. However, our framework differs from DCM in a few important ways:
(i) our underlying framework and model is based on Structural Causal Models rather than dynamical systems, and
(ii) we seek to formalize the notion of information flow, not just of effective connectivity.
However, the style of thinking, which involves starting from theoretical models and incorporating the stimulus and experimental design, is common to both DCM and our approach.
1.4 Outline of the Paper
In this paper, we start by giving a mathematical description of a generic computational system, about which inferences are being drawn (Section 2). We then formally define what it means for information about some message to flow on a single edge or on a set of edges in the computational system (Section 3). This is done by proposing an intuitive property that we would like such flows to satisfy, along with some candidate definitions, and then examining which candidates satisfy the property. The intuitive property we desire is: information flow about a message may not completely disappear from the system at a certain time, only to spontaneously reappear at a later point (formalized in Property 1). It emerges that simple and intuitive definitions actually fail to satisfy this basic property, and so a more sophisticated definition is needed. We then show how our definition for information flow about the message satisfies several desirable properties, including guarantees for the existence of so-called “information paths” between appropriately defined input and output nodes (Section 4). After this, we suggest how one might detect which edges of the computational system have information flow, and provide an “information path algorithm”, which identifies the aforementioned information paths (Section 5). We also introduce and discuss the concepts of derived information, redundant transmissions and hidden nodes, which allow one to obtain a more fine-grained understanding of information structure in the computational system. To show that our definition of information flow agrees with intuition, we give several canonical examples of computational systems and depict the information flow in each case (Section 6). Finally, we conclude with discussions on connections with neuroscience, issues related to the difficulty of estimating information flow (along with possible remedies), comparison with the existing directed causal influence literature, connections with fields such as probabilistic graphical models and causality, and a discussion on information volume (Section 7).
2 The Computational System
Our goal is to develop a rigorous framework for understanding how the information about a message flows in a computational system. To do this, we first need to define the terms “computational system”, “message”, “information about a message” and “flow”. In this section, we start with the first two terms, defining the model of the computational system that is used throughout this paper, and explicitly defining the message.
Our model is based on prior art in the information theory literature [11, 12], and consists of nodes communicating to each other at discrete points in time on a directed graph. At every time instant, each node receives transmissions on its incoming edges and computes a function of these transmissions to send out on its outgoing edges. This function can be random and time-dependent, and can be different for every outgoing edge. We will be interested in the flow of a particular random variable called the “message”, which will be defined shortly. Since the directed graph forming the computational system may have cycles, the message may flow along a cyclic path. To deal with this possibility while capturing the fact that nodes must be causal444Causal in the “Signals and Systems” sense of the word, where a node cannot make use of future transmissions [44]., we define a “time-unrolled” graph (in a manner similar to Ahlswede et al.555Although the work of Ahlswede et al. (2000) is titled “Network Information Flow”, it actually addresses a different problem: one of the achievable rate region of a broadcast network and the optimal coding strategy that achieves this rate. In contrast to their work, which concentrates on characterizing and achieving the optimal rate, our focus is on understanding how information about a known message flows in an existing computational system. [12]), which describes how nodes communicate to each other over time. We define a random variable model for the nodes’ transmissions, and demonstrate how each node computes these variables. We also formally define the input nodes of the computational system, through their relationship with the message.
Definition 1** (Complete directed graph).**
A complete directed graph is described by a set of nodes and the set of all edges between those nodes (including self-edges). We denote the set of nodes by their indices, , where is a positive integer denoting the number of nodes in the graph. The set of edges in the graph is the set of all ordered pairs of nodes, .
Note that
(i) edges are directed, so the edge describes an edge from node to node ; and
(ii) nodes have self-edges. For every , there is an edge in .
Moving forward, nodes shall be thought of as performing computations and possessing local memories. We shall interpret the transmission of a node to itself as the variable it stores within its memory666Instances of directed graphs that are not complete and of nodes possessing no memory are merely special cases of our model, where the respective edges’ transmissions can simply be set to zero..
Definition 2** (Time-unrolled graph).**
In order to allow nodes to have different transmissions at every time instant, we must provide for the progression of time. Let be a set of time indices, where is a positive integer representing the maximum time index. Then, a time-unrolled graph is constructed by indexing a complete directed graph using the time indices as follows:
The nodes consist of all nodes in , subscripted by time indices in ,
[TABLE] 2. 2.
The edges connect nodes of successive times in , so they can be written in terms of the edges in as
[TABLE]
For brevity, we denote the set of all nodes at time by , and the set of all (outgoing) edges at time by . So, for example, we will have and . All of the notation in this section can be visualized in Figure 1 and is summarized in Table 1.
Once again, note that
(i) edges at time connect nodes at time to nodes at time ; and
(ii) since the original graph had self-edges, there will always be an edge in for every node .
Also note, we have only presented the complete directed graph in Definition 1 in order to explicitly define the process of time-unrolling. We do not expect the time-unrolled graph to be “rolled back” into a complete directed graph at the end of an information flow analysis. Since we seek a time-evolving picture of information flow between different computational nodes, we will directly view and interpret information flow on the time-unrolled graph. This is illustrated later, through several examples, in Section 6.
Definition 3** (Computational System).**
A computational system is a time unrolled graph that has transmissions on its edges which are constrained by computations at its nodes. The input to the computational system includes a message777The message is the random variable whose “information flow” we will seek to identify., . We now elaborate upon these terms:
- 3a)
Transmissions on Edges**
We begin by defining a function which maps every edge of to a random variable. Let be the set of all random variables in some probability space888We assume that all probability distributions are such that the mutual information and conditional mutual information between any sets of random variables is well-defined [45, Sec. 2.6].. Then, let be a function that describes what random variable is being transmitted on a given edge, i.e., is the random variable corresponding to the transmission on the edge .
For convenience, we define applied to a set of edges as the set of random variables produced by applying to each of those edges individually, i.e., for any set ,
[TABLE]
We extend the use of this notation to other functions of nodes and edges that we define, going forward. 2. 3b)
Computation at a Node**
Let be a node in the time-unrolled graph , at some time (recall that ). Let be the set of edges entering , and be the set of edges leaving . Further, let us suppose that is able to intrinsically generate the random variable999 and may also be random vectors instead of random variables, i.e., an edge may transmit a vector. This does not affect the theoretical development presented in this paper; all of our proofs remain unchanged.* at time , where W(A_{t})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}W(\mathcal{V}\!\setminus\!\{A_{t}\})\;\forall\;A_{t}\in\mathcal{V}, W(\mathcal{V}_{t})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\{M,X(\mathcal{E}_{t-1})\} and the symbol “\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}” stands for independence between random variables. Then, the computation performed by the node (for ) is a deterministic function101010This kind of model is not new, and can be found in the causality literature for instance, under the name “Structural Equation Models” [13, Sec. 1.4.1]. that satisfies*
[TABLE]
Here, , , , and all make use of the notation described in (1).
Note that the definition above does not apply when ; this is a special case which is discussed below. Also, for convenience, where is an arbitrary set of nodes, we will use to denote the “joint function” mapping the incoming transmissions of all nodes in (along with their intrinsic random variables ) to their respective outgoing transmissions. 3. 3c)
The Message and the Input Nodes**
Each of the nodes in may receive one or more random variables from the world external to the computational system at time . The message, , is simply a specific random variable that is of interest to the experimentalist observing the computational system, and for which we shall define information flow. For now, we assume that we are interested in a single message.111111That is, we assume that the message is a single random variable or vector. It is possible to simultaneously examine the information flows of several (possibly dependent) messages, or of sub-messages within a single message. These cases are examined in Section 5.6. We also assume that the message enters the computational system only at time , and at no later time instant.
We formally define the input nodes of the system as those nodes of , at time , whose transmissions statistically depend on the message :
[TABLE]
where represents the set of edges leaving the node .
To remain consistent with Definition 3b, we define the computation performed by an input node as a function that satisfies
[TABLE]
and the computation performed by a non-input node at time , , as a function that satisfies
[TABLE]
As before, W(A_{0})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}W(\mathcal{V}_{0}\!\setminus\!\{A_{0}\})\;\forall\;A_{0}\in\mathcal{V}_{0} and W(\mathcal{V}_{0})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}M.
Remarks
Informally speaking, Definition 3 is designed to allow each node to generate a randomized function of its incoming transmissions for each of its outgoing transmissions. 2. 2.
The randomization at each node is explicitly captured by its intrinsic random variable , and is assumed to be independent across all nodes of the system. 3. 3.
Furthermore, each node is allowed to send a different transmission on each of its outgoing edges. 4. 4.
Note that the condition imposed by Equation (2) introduces dependence between the random variables in the set . 5. 5.
For the most part, we will not be concerned with the precise form of the computation being performed by every node. We will only make use of information-theoretic measures applied to the message and to the random variables in the computational system.
Throughout the paper, we use the variables , , , , and to refer to nodes and , , , and to refer to edges. We use their script forms, e.g. , when referring to sets of nodes and edges, and primed script forms, e.g. , when referring to subsets thereof. Once again, the notation we use is summarized in Table 1, and depicted in Figure 1 for convenience.
Having defined what we mean by the terms “computational system” and “message”, in the following sections we proceed to find a definition for “information flow” and identify properties that this definition satisfies in any computational system.
3 Defining Information Flow
Before one can speak of detecting information flow in a network, it is first important to define what it is that we seek to detect.121212In essence, “causal influence” measures such as Granger Causality and Directed Information, while intuitively quantifying transferred information, fail to lay down what aspect of computation they actually capture. This is, in part, a result of conflating the stages of defining a quantity we want to understand, and prescribing an estimator for it. In this section, we focus on arriving at a definition for information flow.
Our goal is to formalize how information about a message flows in a computational system. Ultimately, we expect to find the path that the message takes while being processed by the system. Towards this, we start by trying to formally define what it means for information about the message to flow on a given edge. This section concludes with a proposal for such a definition: one based on strict positivity of a conditional mutual information. But to provide the intuition behind this choice of definition, we start with several simpler candidate definitions, and show how they fail to satisfy an intuitive property using counterexamples.
After proposing a definition for information flow, in Section 4, we discuss the properties satisfied by our definition. Then, in Section 5, we specify how the transmissions of the computational system are observed, and describe how information flow might be inferred in a real computational system.
3.1 An intuitive property
To concretely define what it means for information about a message to flow on an edge, we need some way to assess competing candidate definitions and choose one among them. Towards this goal, we state a straightforward and intuitive property, which we would want any definition of information flow to satisfy.
Suppose that, at a given point in time, there is no flow of information about the message across any edge of a computational system. Note that this includes self-edges, so no node “carries” information about the message within its memory either. Then, we expect that information about the message has ceased to persist in the system, so the information flow about the message must be zero on all edges of the computational system, at all future points in time.
Property 1** (The Broken Telephone131313https://en.wikipedia.org/wiki/Telephone_game).**
Let be a computational system, and let be an indicator of the presence of information flow about on an edge. That is, , if information about flows on the edge and , otherwise. The Broken Telephone Property states that if, at some time , we have
[TABLE]
3.2 Intuiting Information Flow through Counterexamples
We now propose four candidate definitions, beginning with the simplest. We then construct counterexamples to show how the first three candidate definitions do not satisfy Property 1.
Candidate Definition 1**.**
A simplistic and intuitive definition for information flow might simply stem from dependence. We say that information about the message flows on an edge if
[TABLE]
Counterexample 1**.**
Consider the computational system depicted in Figure 2 (note that, in order to avoid unnecessary clutter, only edges with non-zero transmissions are shown in the figure). is the input node, which has the message at time . The system’s goal is to communicate141414This communication can be thought of as computing the identity function, and making the output available at the node . to the node . It chooses the following strategy: at , “transmits” to (i.e., node stores in its memory). independently generates a different random number, , Z\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}M, and sends this message to , while also storing it in memory it until . then computes and passes the result to , while sends to . Here, the symbol “” stands for xor, the exclusive-or operator on two bits. is thus able to recover by once again xor-ing its inputs, and .
Note that the output of depends on , even though none of its inputs individually depends on . That is, I\bigl{(}M;X((A_{1},B_{2}))\bigr{)}=I(M;M\oplus Z)=0, and I\bigl{(}M;X((C_{1},B_{2}))\bigr{)}=I(M;Z)=0, so by Candidate Definition 1, information about the message flows on no edge at time . However, information about the message does flow out of node at time . This violates Property 1. Thus, mere dependence on the message cannot be a valid definition for flow of information on a single edge.
Communication strategies such as the one in Counterexample 1 frequently arise in cryptography [46], to prevent an eavesdropper from reading confidential information, and in network coding [12], for achieving the communication capacity of a network. Furthermore, a complex computational network may have smaller sub-networks with such topologies. For instance, we observe such a sub-network in the canonical example for network coding: the butterfly network [12, Fig. 7b] (this particular example is discussed in detail in Section 6.1). Optimal communication in such a network requires the use of such topologies, so Counterexample 1 is far from obscure. In fact, central to the idea of Counterexample 1 is a concept known as “synergy”, which is well-studied in the literature on Partial Information Decomposition [47, 48, 49] (see [50] for a recent review). This is discussed at length in Section 3.5. Even in neuroscience, the concept of synergy is recognized and well-understood [51, 52, 53], and some experimental evidence has appeared in the literature [54].
Counterexample 1 demonstrates that the information necessary to recover the message (or a function of it) is not necessarily transmitted through individual edges, but jointly across edges. So, we might instead seek to define the “smallest set of edges” along which information about the message flows, for every point in time. But if we ultimately wish to isolate paths along which information about the message flows, we require an understanding of which edges specifically the information flows upon. We therefore continue to think of information as flowing on individual edges.151515It should be noted that the two views—information flowing on individual edges, versus sets of edges—are compatible with each other if we use Definition 5 (which will appear shortly) to describe information flow on a set of edges. This equivalence is elaborated upon in Section 3.4. Later, in Section 4.4, we attempt to refine our understanding of the aforementioned “smallest set of edges” along which information about the message flows.
We can now update our naïve definition to counter the previous counterexample. We start by noting that in Counterexample 1, although the transmission on edge is independent of , it is not conditionally independent of when given the transmission on .
Candidate Definition 2**.**
We say that information about the message flows on an edge if one of the following holds:
I\bigl{(}M;X(E_{t})\bigr{)}>0, or 2. 2.
* s.t. I\bigl{(}M;X(E_{t})\,\big{|}\,X(E_{t}^{\prime})\bigr{)}>0.*
Counterexample 2**.**
Consider a modified version of Counterexample 1, shown in Figure 3. Now, since there are two noise terms, no single extra edge may be conditioned upon to have non-zero information flow at time . So, Candidate Definition 2 also fails to satisfy Property 1.
It might seem that a possible rectification is to condition on all other edges at time , but we can show that this also fails the test.
Candidate Definition 3**.**
We say that information about the message flows on an edge if one of the following holds:
I\bigl{(}M;X(E_{t})\bigr{)}>0, or 2. 2.
I\bigl{(}M;X(E_{t})\,\big{|}\,X(\mathcal{E}_{t}\!\setminus\!\{E_{t}\})\bigr{)}>0.
Counterexample 3**.**
Consider the computational system shown in Figure 4. Once again, we have an input node which possesses the message at time , and wishes to send this message to node . It does so by mixing with an independent random variable generated at , so that the scenario described in Counterexample 1 still holds. But additionally, communicates to along a redundant path, through . Now, if is any incoming edge of , it is still true that I\bigl{(}M;X(E)\bigr{)}=0. So none of the inputs of individually depends on , thus eliminating the first condition in Candidate Definition 3. Furthermore, checking each incoming edge of reveals that the second condition also fails to hold. If we take , we get
[TABLE]
The same holds true when since the transmissions on both edges are identical by construction. Likewise, if we take , we have
[TABLE]
with the same holding true when . Therefore, no edge at time has any information flow about the message , as per Candidate Definition 3. Nevertheless, is able to recover the message at time , proving that Property 1 fails to hold for Candidate Definition 3.
3.3 Information Flow on a Single Edge
The counterexamples presented in the previous section motivate a new definition for when information about the message can be said to flow on a given edge. Neither dependence of on the transmission of an edge, nor conditional dependence given one or all other edges, satisfy Property 1.
However, in all these counterexamples, given an edge upon which we expect to have non-zero information flow, we observe: there is at least one subset of edges , such that when given , is conditionally dependent161616Equivalently, we could say that there exists at least one subset of edges , without explicitly excluding , since I\bigl{(}M;X(E_{t})\,\big{|}\,X(E_{t}),X(\mathcal{E}_{t}^{\prime})\bigr{)}=0. on . In Counterexample 1, the edge , carrying , is conditionally dependent on , given X\bigl{(}(C_{1},B_{2})\bigr{)}=Z. In Counterexample 2, X\bigl{(}(A_{1},B_{2})\bigr{)}=M\oplus Z_{1}\oplus Z_{2} is conditionally dependent on , given \{X\bigl{(}(C_{1},B_{2})\bigr{)},X\bigl{(}(D_{1},B_{2})\bigr{)}\}=\{Z_{1},Z_{2}\}. And finally, in Counterexample 3, X\bigl{(}(A_{1},B_{2})\bigr{)}=M\oplus Z is conditionally dependent on , given X\bigl{(}(C_{1},B_{2})\bigr{)}=Z; note that we do not condition on X\bigl{(}(D_{1},B_{2})\bigr{)}=M\oplus Z. Thus, conditioning on a subset of the other edges’ transmissions creates dependence between and the transmission on an edge of interest.
We will shortly prove that Property 1 holds when information flow is defined as below, so we directly state it as a definition, skipping its candidacy status.
Definition 4** (-information Flow on a Single Edge).**
We say that information about the message flows on an edge if
[TABLE]
Henceforth, we refer to “information flow about the message ” as -information flow, and use the phrase “the edge has -information flow” or “the edge carries -information flow” to mean that information about flows on per this definition.
Note that if I\bigl{(}M;X(E_{t})\,\big{|}\,X(\mathcal{E}_{t}^{\prime})\bigr{)}>0, then I\bigl{(}M;X(\{E_{t}\}\cup\mathcal{E}_{t}^{\prime})\bigr{)}>0. In other words, there exists a set of edges that includes , whose transmissions depend on . This is why it is important to condition on all possible subsets of . It is not immediately clear, however, whether every edge in has -information flow. We return to this point in Section 4.4.
Also, this definition implies that certain edges, such as in Counterexample 1, may have -information flow, which may seem counter-intuitive. This is discussed further and justified in Section 4.2.
3.4 Information Flow on a Set of Edges
The definition of -information flow for a single edge naturally generalizes to one for a set of edges, at a given time.
Definition 5** (-information Flow on a Set of Edges).**
We say that information about the message flows on a set of edges if
[TABLE]
The definition of -information flow on a set of edges is nearly identical to its single-edge counterpart. Indeed, they are closely related, as the following proposition shows.
Proposition 1**.**
A set has -information flow if and only if there exists an edge that has -information flow.
A proof of this proposition can be found in Appendix A.
It should be noted that although the counterexamples in this section all employed computational systems which recovered the message at a new node at a later time, a computational system will in general compute some function of the message. For instance, see the example in Section 6.2.
3.5 The Connection with Synergistic Information
This section connects our definition of -information flow with recent developments on a subject known as “Partial Information Decomposition” (PID). Our definition is closely related to the concept of “Synergistic Information” that appears in this field. This section exists only for the purpose of providing a deeper intuition for our definition of -information flow, and does not affect the rest of the paper in any significant way. We have attempted to explain this intuition in a way that is accessible to readers unfamiliar the PID literature. However, readers may feel free to skip this section, if desired.
At its core, Counterexample 1 relies on a concept known as “synergy”, which is described explicitly in the literature on Partial Information Decomposition (PID) [47, 48, 49] (see [50] for a recent review, and Appendix C for a brief introduction). Essentially, this body of literature seeks to decompose the mutual information that two or more variables share about a message, I\bigl{(}M;(Y_{1},Y_{2},\ldots)\bigr{)}, into several individually meaningful, non-negative components. In particular, when discussing the bivariate case—i.e., the case of two variables, I\bigl{(}M;(Y_{1},Y_{2})\bigr{)}—it is understood what the terms in this decomposition should be:
(i) information about the message that each variable carries uniquely, and which cannot be inferred from the other;
(ii) information about the message that the variables share redundantly, and which can be extracted from either;
(iii) and information about the message that the variables convey synergistically, which is revealed only when both variables are taken together, and cannot be inferred from either variable individually.
Counterexample 1 is the canonical example for synergy, and is known simply as the “xor” example in the PID literature. While and are individually independent of , when taken together, I\bigl{(}M;(M\oplus Z,Z)\bigr{)}=H(M). This suggests that and have no unique or shared information about , but convey information synergistically.
While the field has not yet arrived at a consensus on the most appropriate definitions for unique, redundant and synergistic information [50], it is well-understood what properties these quantities must satisfy, at least in the bivariate case (see Appendix C, specifically, Equations (94), (95) and (97)). Therefore, even without formal definitions, we can rely on the intuition provided by these properties to understand the implications of PID for -information flow. If a particular edge’s transmission contains unique or redundant information about the message (with respect to some other subset of edges at that point in time), then that information will manifest itself in the form of strictly positive mutual information. However, in the absence of positive mutual information between the message and the transmission on a given edge, we need to consider whether said transmission synergistically interacts with another subset of transmissions at that point in time, as this could potentially create dependence with the message through the kind of “recombination” described in Counterexample 1. We then need to decide whether such synergistic interactions ought to be considered to constitute information flow. As we show below, our definition of -information flow does consider instances of purely synergistic information to constitute information flow.
Indeed, it is possible to formulate a definition for information flow based on synergy, which is completely equivalent to Definition 4. The definition below makes use of the PID preliminaries given in Appendix C.
Definition 6** (-synergistic information flow).**
We say that an edge has -synergistic information flow if at least one of the following holds:
I\bigl{(}M;X(E_{t})\bigr{)}>0, or 2. 2.
\exists\;\mathcal{E}_{t}^{\prime}\subseteq\mathcal{E}_{t}\setminus\{E_{t}\}\quad\text{s.t.}\quad CI\bigl{(}M:X(E_{t});X(\mathcal{E}_{t}^{\prime})\bigr{)}>0,**
where represents the synergistic information between and about .
Proposition 2** (Equivalence of Information Flow Definitions).**
An edge has -information flow if and only if it has -synergistic information flow. Furthermore, suppose is an edge which satisfies I\bigl{(}M;X(E_{t})\bigr{)}=0. Then,
[TABLE]
for some set , if and only if
[TABLE]
That is, the set upon whose transmissions we need to condition is the same as the one responsible for providing synergy in the alternate definition.
A proof of this proposition is given in Appendix C.
We should also mention here that it may be possible to leverage the more recent definitions of synergy to supply an intuitive measure of the volume of information flow; we discuss this in Section 7.5.
4 Properties of Information Flow
Having defined what it means for information about a message to flow on an edge, we demonstrate that Definition 4 satisfies several intuitively desirable properties, including Property 1.
4.1 The Broken Telephone Property
Theorem 3**.**
-information flow, as given by Definition 4, satisfies Property 1.
Before we prove this theorem, we prove a simpler lemma which directly falls out of Definition 4 and the properties of mutual information.
Lemma 4**.**
There is no edge in that carries -information flow if, and only if, is independent of . In other words,
[TABLE]
Equivalently, we can state the opposite: depends on if and only if at least one edge in carries -information flow.
Proof.
() Suppose that the condition in (14) holds. Let \mathcal{E}_{t}=\bigl{\{}E^{(1)}_{t},E^{(2)}_{t},\ldots,E^{(N^{2})}_{t}\bigr{\}} be any ordering of the edges in . Then,
[TABLE]
where (a) follows from the chain-rule of mutual information [55, Ch. 2], (b) is simply the application of Equation (1), and (c) follows from the fact that each term in the summation is zero, by (14). This proves the forward implication.
() Next, suppose I\bigl{(}M;X(\mathcal{E}_{t})\bigr{)}=0. Let be any edge in and let be any subset of . Also, let \mathcal{E}_{t}^{\prime\prime}=\mathcal{E}_{t}\setminus\bigl{(}\mathcal{E}_{t}^{\prime}\cup\{E_{t}\}\bigr{)}. Then,
[TABLE]
by the chain rule. Since (conditional) mutual information is always non-negative [55, Ch. 2], all three terms on the right hand side must be zero. So in particular,
[TABLE]
Since and are arbitrary, this proves the converse. ∎
Proof of Theorem 3.
We need to prove that -information flow, as given by Definition 4, satisfies Property 1. Explicitly stated, we need to show that if every edge at some time has zero -information flow, then every edge at all future times must also have zero -information flow. So suppose that, at time , for every we have
[TABLE]
By Lemma 4, this implies that
[TABLE]
Now, consider the first future time instant, . For every node , the definition of computation at a node (Definition 3b) states that
[TABLE]
where the reader may recall, and are the edges entering and leaving respectively. We can collect the individual functions across all nodes in into a single joint function , as described in Definition 3b, to obtain
[TABLE]
Therefore,
[TABLE]
where (a) follows from the Data Processing Inequality [55, Ch. 2], (b) follows from the fact that W(\mathcal{V}_{t+1})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\{M,X(\mathcal{E}_{t})\}, and (c) follows from (23). Once again, by non-negativity of mutual information we must have that {I\bigl{(}M;X(\mathcal{E}_{t+1})\bigr{)}=0}. Applying Lemma 4 once again, we find that for ,
[TABLE]
We have shown that (22) implies (30), so induction on yields that (30) holds for all future times , completing the proof. ∎
4.2 The Existence of Orphans
Definition 4 also has a very non-intuitive property: an edge leading out of a node may have -information flow, even though no edge leading into that node has -information flow.
Definition 7** (-information Orphan).**
In a computational system , a node is said to be an -information orphan if has -information flow (as per Definition 5), but has no -information flow.
Property 2**.**
-information orphans may exist in a computational system.
Proof.
Consider the computational system in Figure 2 from Counterexample 1. The node is an -information orphan, since the edge carries -information flow, whereas none of its incoming edges carries -information flow. ∎
The existence of -information orphans, along with the presence of -information flow on in Counterexample 1, may not be expected, since was never computed from . Indeed, -information flow appears to emerge from “nowhere” at the node , leaving nodes such as orphaned in a view of the graph that contains only edges having -information flow, and hence the name. But closer inspection reveals that in this example, the transmissions arriving at from and , i.e. and , are statistically identical: they are both individually independent of , but when xor’ed, are fully dependent on . In other words, any purely observational measure171717i.e., a functional of the joint distribution of defined on the transmissions at time that assigns -information flow to , must also assign -information flow to .
Note that, just as -information flow can originate at an -information orphan, -information flow may also terminate at a node—either by simple omission, or as a result of some computation (see Section 6 for such instances). Likewise, multiple outgoing edges of a given node may transmit redundant copies of the same information. Ultimately, we see that there is no “law of conservation” for -information flow. In this sense, “information flow” is not a typical kind of “flow” that is defined on graphs (see, for example, [56, Sec. 26.1]), and well-known results such as the Max-flow Min-cut Theorem [56, Thm. 26.6] do not apply as-is to -information flow.
It is worthwhile to note at this point that the existence of -information orphans such as in Counterexample 1 is not inconsistent with the Data Processing Inequality [55, Ch. 2]. In fact, a clear example of the Data Processing Inequality in play is seen at the network-level, wherein —— form a Markov Chain for any time , and so the information content about present collectively in all transmissions at time must be no more than that present at time . We call this Global Markovity, and state it formally for completeness.
Corollary 5** (Global Markovity).**
At any given time , the following Markov Chain holds: ——.
In fact, this Markov condition must hold for every subset of nodes, not just for the entire set of nodes, so it is subsumed by the following proposition.
Proposition 6** (Local Markovity).**
For any given subset of nodes , the following Markov Chain holds: ——.
Proof.
Since X(\mathcal{Q}(\mathcal{V}_{t}^{\prime}))=f_{\mathcal{V}_{t}^{\prime}}\bigl{(}X(\mathcal{P}(\mathcal{V}_{t}^{\prime})),W(\mathcal{V}_{t}^{\prime})\bigr{)} by Definition 3b, the tuple \bigl{(}X(\mathcal{P}(\mathcal{V}_{t}^{\prime})),X(\mathcal{Q}(\mathcal{V}_{t}^{\prime}))\bigr{)} is also a function of and . Hence, the following Markov chain holds:
[TABLE]
By the Data Processing Inequality, this implies that
[TABLE]
where in (a) and (b), we have used the chain rule of mutual information in two different ways, and in (c) we have used the fact that W(\mathcal{V}_{t}^{\prime})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\{M,X(\mathcal{P}(\mathcal{V}_{t}^{\prime}))\}. Therefore,
[TABLE]
which implies the Markov chain in Proposition 6. ∎
Since the above also holds for , wherein , Proposition 6 implies Corollary 5.
Given that these Markov conditions arise directly from the way we have defined the computational system, specifically Definition 3b, they may not be very surprising (indeed, they may be considered properties of the computational system model itself). However, it is worth noting that Proposition 6 holds even at an -information orphan. Thus, -information orphans do not “create” information about , as we would rightly expect, given the Data Processing Inequality.
4.3 The Existence of Information Paths
We now show that if the outgoing transmissions of any given node depend on the message, then we can find a path leading to that node from one or more input nodes, along which -information flows. Before we demonstrate this property, we formally define what we mean by the terms “path” and “cut”.
Definition 8** (Path).**
*In any computational system , suppose and are two disjoint sets of nodes in . Then, a path from to is any ordered set of nodes that satisfies
(i) ;
(ii) ; and
(iii) for every ,
where is a positive integer indicating the length of the path. We refer to the set as the edges of the path.*
Definition 9** (-Information Path).**
Continuing from Definition 8, we define an -information path from to as any path from to , each of whose edges carries -information flow. That is, if for some , then for every ,
[TABLE]
Definition 10** (Cut).**
*In any computational system , suppose and are two disjoint sets of nodes in . Then, a cut separating and is any pair of sets , such that
(i) ;
(ii) ;
(iii) ; and
(iv) .
We refer to the set of edges going from to , i.e. , as the edges in the cut set181818Note that it is not necessary for us to assume that, individually, and are connected sets of nodes. For instance, there may be an isolated subset of , surrounded only by nodes in . Our theorems and proofs remain unaffected, even in such a scenario..*
Definition 11** (Zero–-information Cut).**
Continuing from Definition 10, we say that a cut is a zero–-information cut if every edge in its cut set has zero -information flow. That is, for every ,
[TABLE]
Remark
In Definition 11, we require that Equation (37) hold for every edge in . However, the edges in this set may belong to several different time points, since the cut is not restricted to any particular time (e.g., see Figure 5). The time used in Equation (37), therefore, is determined by the time of the edge , and varies for each that we check in .
Property 3** (Existence of an Information Path).**
In any computational system , suppose that at some time , there is an “output node” whose outgoing edges satisfy I\bigl{(}M;X(\mathcal{Q}(V_{\text{op}}))\bigr{)}>0. Then, there must exist an -information path from the input nodes to .
Theorem 7**.**
Definition 4 satisfies Property 3.
While the theorem seems obvious on the surface, the proof is in fact non-trivial because of the nature of our definition of -information flow. Due to Property 2, -information flowing out of a node does not imply that -information must flow into that node. Therefore, a straightforward application of the Data Processing Inequality at every node fails to prove the theorem, and we must resort to a more rigorous cut-set-based approach.
Proof outline.
We shall prove the contrapositive of the theorem, i.e., we will show that if there exists no -information path from to , then the outgoing transmissions of are independent of . We first connect the absence of any -information path with the presence of a zero–-information cut. This is achieved in Lemma 8, which we present before the proof of Theorem 7.
The proof itself proceeds by induction over time. We divide the proof into two steps: initialization and continuation. Starting with the first nodes that come after the cut (temporally) in the initialization step, we systematically show that all nodes to the right of the cut have outgoing transmissions that are independent of the message through induction. In this proof outline, we show these steps intuitively using Figure 5, where the dashed black line denotes the cut.
Initialization. Here, node is the first node to the right of the cut, and all of its incoming edges must come from across the cut (depicted by lines in red). Because the cut is a zero–-information cut, none of its incoming transmissions have -information flow. Furthermore, the intrinsically generated random variable is independent of . Using these two facts along with the Data Processing Inequality, we can show that the transmissions on ’s outgoing edges, , are also independent of .
Continuation. At the second time instant to the right of the cut, nodes and receive their incoming transmissions from either (shown in orange) or from across the cut (shown in blue). Once again, the transmissions coming from across the cut can have no information flow, and we have shown that the transmissions coming from are independent of . Also, and are independent of and all incoming transmissions. This suffices to show that the outgoing transmissions of and , X\bigl{(}\mathcal{Q}(B_{2})\cup\mathcal{Q}(C_{2})\bigr{)}, are independent of . Applying this argument repeatedly over time shows that the transmissions of all nodes to the right of the cut are independent of .
Therefore, if there is a node whose outputs depend on , we can be assured that there exists no zero–-information cut separating from . Therefore, by Lemma 8, there exists an -information path from to . ∎
A few nuances are omitted in this outline, such as how the definition of plays a role precisely. These subtleties are better elucidated in the full proof.
Before proceeding to the formal proof of Theorem 7, we first state and prove the lemma we alluded to earlier, which shows how the absence of an -information path implies the presence of a zero–-information cut, and vice versa.
Lemma 8**.**
Let and be two disjoint sets of nodes in the computational system . There exists no -information path from to if and only if there is a zero–-information cut separating and .
Proof.
() Suppose there exists no -information path from to . Consider the set of all nodes to which there exists at least one -information path from . Let be the collection of all such nodes, along with the nodes in , i.e.,
[TABLE]
Let , so that consists of nodes to which there is no -information path from . Then, we must have , since it is known that there are no -information paths from to . Therefore, is a cut that separates and , such that no edge in the cut set has -information flow. In other words, by Definition 11, this is a zero–-information cut separating and .
() Next, suppose that there is an -information path from to . Then, we claim that there can exist no zero–-information cut separating and . Let be any cut separating and . By Definition 8, we must have and . So, there must be at least one edge going from to which lies on the path. This implies that at least one edge in the cut set carries -information flow. Since the conditions of Definition 11 are not satisfied, this cut is not a zero–-information cut. Since this is true for every cut separating and , the claim holds. ∎
Proof of Theorem 7.
As mentioned in the proof outline, we prove the contrapositive of the theorem. Suppose there exists no -information path from the input nodes to . Then, by Lemma 8, there exists a zero–-information cut191919Note that, in general, this cut may be arbitrarily complex, spanning several nodes and multiple time instants. separating and . We use this to prove that the transmissions of are independent of .
Setup. Let the cut separating and be given by , so that and . Then, the cut divides into the following sets: , the edges between the nodes in ; , the edges between nodes in ; and , the edges going from to (the edges going from to will not be relevant to our discussion). From the previous paragraph, Lemma 8 implies that is a zero–-information cut, so by Definition 11, we have that for all ,
[TABLE]
Note that the edges in may belong to different time instants. In particular, the time instant in the equation above corresponds to the time of the edge , whose flow is in question.202020In fact, this is one of the central factors that prevents us from recursively applying the Data Processing Inequality at every node, leading from to .
Order the nodes in by time, and let be the subset of nodes in at time . Let and respectively be the sets of edges collectively entering and leaving all nodes in . We shall prove that the outgoing transmissions of every node in , including those of , must be independent of the message, i.e.,
[TABLE]
Initialization. Let be the first time instant for which is non-empty. Then, we encounter two cases: either , in which case the nodes in have no incoming edges, or , and the nodes in have incoming edges. We shall first prove that in both cases, the outgoing transmissions of are independent of the message, i.e. I\bigl{(}M;X(\mathcal{Q}(\mathcal{V}^{\text{sink}}_{t_{0}}))\bigr{)}=0.
(Case I) When , . This is because the cut separates from , with , so no nodes in can be input nodes. So, by the definition of (non-)input nodes (Definition 3c), we must have
[TABLE]
where step (a) uses the data processing inequality and step (b) makes use of the fact that W(\mathcal{V}_{0})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}M.
(Case II) When , the definition of implies that all nodes at time are in , so all incoming edges of must lie in the cut set, i.e., . Since the cut is a zero–-information cut, we have that for all ,
[TABLE]
Once again, considering , we have
[TABLE]
where (a) and (b) follow from the Data Processing Inequality and the chain rule of mutual information respectively. In step (c), the first expression in the sum goes to zero by taking in (45) and the second expression is zero since W(\mathcal{V}^{\text{sink}}_{t_{0}})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\{M,X(\mathcal{E}_{t_{0}-1})\}, and (refer Definition 3b). So, from equations (43) and (49), we have that for all values of ,
[TABLE]
Continuation. Now, suppose that for some , we have I\bigl{(}M;X(\mathcal{Q}(\mathcal{V}^{\text{sink}}_{t-1}))\bigr{)}=0. We shall prove that this implies I\bigl{(}M;X(\mathcal{Q}(\mathcal{V}^{\text{sink}}_{t}))\bigr{)}=0. First, observe that
[TABLE]
For convenience, let and . We have used the subscript here to remind the reader that , which are the incoming edges of , are a subset of . Then, we have
[TABLE]
Since the cut is a zero–-information cut, we have that for every ,
[TABLE]
Therefore, by Definition 5 and Proposition 1,
[TABLE]
Secondly, . This is depicted in Figure 5, and explained in the caption. So,
[TABLE]
where (a) follows from the fact that considering more random variables can only increase mutual information, and (b) follows from the induction assumption. Finally, consider how depends on :
[TABLE]
where once again, (a) and (b) follow from the data processing inequality and the chain rule respectively. In step (c), the first and second terms go to zero by equations (56) and (54) respectively, while the third term is zero since W(\mathcal{V}^{\text{sink}}_{t})\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\{M,X(\mathcal{E}_{t-1})\} and .
The proof follows from induction on , so
[TABLE]
If there exists an output node whose transmissions depend on , then there can exist no cut consisting of edges with zero -information flow, and hence by Lemma 8, there must be a path consisting of edges that carry -information flow between the input nodes and the output node in question. ∎
4.4 The Separability Property
Finally, we state a property that may be of interest to obtain a deeper understanding of the nature of -information flow, as given by Definitions 4 and 5.
Proposition 9** (Separability).**
Let be a computational system. Then, at any given point in time , there exist two sets , such that all of the following conditions hold:
** 2. 2.
** 3. 3.
Either , or for every there exists a subset such that
[TABLE] 4. 4.
Either , or for every ,
[TABLE]
A proof of this proposition can be found in Appendix B.
Proposition 9 shows that at any given point in time , it is possible to partition into two sets: , consisting only of edges that have -information flow, and , comprising edges that have no -information flow. Furthermore, when considering the -information flow of edges in , it suffices to condition on the transmissions of edges within to ascertain the presence of -information flow. Conditioning upon the transmissions of edges in will not change the mutual information between the message and the transmissions of edges in .
5 Inferring Information Flow
Having discussed the definition and the properties of -information flow, we now consider how these flows of information might be inferred in a real computational system. We first discuss an observation model that describes which random variables are observed and how they are sampled. Under this model, we show how existing techniques from the literature can be used to identify which edges carry -information flow. As in previous sections, we restrict our attention to detecting whether or not a given edge has -information flow, relegating quantification of these flows to future work. Quantification is briefly discussed in the form of an example in Section 6.3, and again in Section 7.5.
We then describe an algorithm that recovers all -information paths between the input nodes and a given output node, by leveraging the knowledge of which edges have -information flow. We also explain how one might attain a fine-grained characterization of the structure of information flow, by introducing the concept of “derived information”. This is useful for understanding which transmissions are “derived” from others, allowing one to find transmissions that are redundant and discover the presence of hidden nodes. Finally, we explain how flows of information about multiple messages can be inferred in our framework.
5.1 The Observation Model
Before we can describe how information flow and information paths can be identified, we must provide a statistical description of the random variables that are observed. Let be a computational system under observation. We then make the following assumptions:
Transmissions on all edges, including self-edges, are observed. The random variables that are intrinsically generated at each node are not observed, unless they are also transmitted on an edge (which could be a self-edge). 2. 2.
Several trials212121The word “trial” is borrowed from the neuroscience literature, wherein a neuroscientist will often conduct multiple trials in a single experiment. In each trial, a human participant or an animal under study is presented with one of a set of carefully chosen stimuli (corresponding to a realization of the message in our setting), and neural activity is recorded using some modality. Scientific inferences are then drawn by making use of the activity from all trials. are observed, each of which corresponds to an independent realization of all random variables in the model222222In reality, trials are not independent in neuroscientific experiments. Indeed, neurons are known to “adapt” their responses from trial to trial, often showing suppressed activity when presented the same stimulus multiple times. This, in part, is considered to be evidence of learning in neural circuitry. However, for simplicity, we restrict our attention here to computational systems that do not learn or show trial-to-trial adaptation.. Every trial uses a realization of which is independently drawn from a distribution determined by the experimentalist232323A more detailed discussion of this distribution can be found in Section 5.6.. For every node , the intrinsically generated random variable is also assumed to be independently and identically distributed across trials. 3. 3.
Observations are made noiselessly, in that the realization of each transmission in every trial is observed as-is, without being further corrupted by random noise of any kind. The implications of noisy measurements will be the subject of future work.
Under these conditions, we discuss statistical tests for information flow that are consistent in the asymptotic limit of infinite trials. It should be noted that these assumptions may be valid to varying degrees in different contexts. This is discussed further in Section 7.1.
5.2 Detecting Information Flow
Given a sample of all random variables described in the observation model, our next task is to identify which edges have -information flow. In other words, we need to describe how the conditions given by Definition 4 can be rigorously tested, and how we might assert with some confidence that a certain set of edges has information flow at each point in time.
According to Definition 4, in order to check whether a particular edge carries -information flow at time , we need to test whether at least one of several conditional mutual information quantities is strictly positive. The standard statistical approach for solving this problem is to frame it as a set of “hypothesis tests”, which in this case is a set of “conditional independence tests”. In general, a hypothesis test formalizes the problem of making an informed decision about the value of some functional of a joint distribution, when observing a sample of data from it. A good conditional independence testing procedure will seek to maximize “statistical power”, i.e. the probability of correctly identifying the presence of conditional dependence, while keeping the probability of an incorrect identification fixed below some “level” that is picked beforehand. One intuitive way to do this might be to construct an estimator for the appropriate conditional mutual information, and “reject” the “null” hypothesis of conditional independence if the conditional mutual information was sufficiently larger than some threshold, . This threshold would have to be chosen so that, on average, the probability of falsely rejecting the null hypothesis is at most . However, there are usually better ways of performing this test, i.e., it is often possible to attain higher power at the same level without actually estimating the conditional mutual information.
While it would be impossible to provide a comprehensive list of papers that have researched the problem of conditional independence testing, it has received (and continues to receive) much attention in the statistics, causality, and information theory communities [57, 58, 59, 60, 61, 62]. In its most general form, conditional independence testing is considered to be a hard problem for continuous random variables [63]. However, if we ignore issues associated with the practical difficulty of estimation (discussed later in Section 7.2), these works provide consistent tests under reasonable assumptions on the joint distribution of the variables involved [59, 60, 61].
Although we mentioned that there are better ways to test for conditional dependence than to estimate the conditional mutual information, there may be instances when one might want to estimate the conditional mutual information anyway. For instance, in an example that will appear shortly in Section 6.3, we rely on an estimate of the conditional mutual information to quantify the amount of -information flowing on a given edge. While our paper has only defined -information flow in terms of whether or not it is present at an edge , it is also extremely useful to know how much -information flow there is. We defer further discussion of this topic until Sections 6.3 and 7.5. For now, we note that several papers have considered how to estimate mutual information and conditional mutual information, both of which might be essential for an understanding of quantification of -information flow [64, 65, 66, 67].
For completeness, we now present a description of how we expect information flow will be detected in practice. We assume that we have samples of observations from every edge of the computational system, at every point in time. If not, appropriate assumptions may need to be made, as discussed later in Section 7.1. At every instant of time , consider the set of all edges present in the network. For every edge , use the following process to determine whether it has -information flow:
First test whether the mutual information between its transmission and the message is greater than zero, i.e., I\bigl{(}M;X(E_{t})\bigr{)}>0. If so, declare that has -information flow. 2. 2.
If not, test for conditional dependence between its transmission and the message, given each of the other edges , i.e., I\bigl{(}M;X(E_{t})\,\big{|}\,X(E_{t}^{\prime})\bigr{)}>0, . If any of these tests rejects the null hypothesis, declare that has -information flow. 3. 3.
If not, test for conditional dependence between and , given subsets of other edges, while sequentially taking edges taken pairwise, then in threes, etc. If any of these tests rejects the null, declare that has -information flow. 4. 4.
If none of the above tests rejects the null hypothesis, declare that carries no -information flow.
Note that we have not discussed the level, , at which we should reject the null in each of the above tests. In general, since we are performing multiple hypothesis tests simultaneously, some manner of “correction” is required to ensure that we do not find, what is effectively, a spurious correlation. This is discussed at length in Section 7.2.
5.3 Discovering Information Paths
Next, we discuss an algorithm that discovers all -information paths leading from the input nodes to a given output node, , in any computational system. As discussed in Section 4, whenever the transmissions of the output node depend on the message, Theorem 7 guarantees that at least one -information path exists.
Algorithm 1, which we propose for recovering all -information paths, is an adaptation of the well-known Depth-First Search242424It is also possible to discover all -information paths using an adaptation of Breadth-First Search [56, Sec. 22.2], but doing so would require some mechanism to prune -information paths that do not lead to the input nodes . So we prefer to use Depth-First Search for simplicity of exposition. method [56, Sec. 22.3]. It takes as its input a computational system in which all edges having -information flow have been identified, the output node , and an empty graph that is completely devoid of nodes and edges. The algorithm returns the set of all -information paths in the form of a directed subgraph of the time-unrolled graph . Starting from , following any path in will lead one to , provided at least one -information path exists.
The algorithm works by recursively visiting nodes, starting from the output node . It traverses only edges that carry -information flow, and uses a marking scheme to avoid revisiting nodes. The same marking scheme is also used to designate nodes to which there are -information paths from . As the algorithm passes through each node, it marks the node “valid” whenever an -information path exists between and that node. If no such path exists, then the node is marked “invalid”. The objective of the algorithm, therefore, reduces to one of finding a path of “valid” nodes from to . The algorithm’s recursive function can be expressed as follows: A node is “valid” if and only if there exists a node such that is valid, and the edge has -information flow. This is a recursive expression since checking the validity of a node at time involves finding valid nodes at time . The only nodes that are considered valid by default are the input nodes .
The algorithm sequentially checks the validity of nodes , starting from the output node . The function FindInfoPaths, when called on any given node , checks the validity of . This involves checking each of the incoming edges of for -information flow. If is a node from which -information flows to , then the algorithm immediately checks the validity of by calling the function FindInfoPaths again. Eventually, if in this recursive process, we arrive at an input node in , then that node is marked “valid”, and added to the output subgraph . Once every node from which -information flows to has been marked “valid” or “invalid”, the validity of can be ascertained. For every “valid” node from which -information flows to , the edge and the node are added to the output subgraph , and is marked “valid”. If there are no such nodes leading to , then is marked “invalid” and does not fall on an -information path.
This recursive logic yields the set of all -information paths leading from the input nodes to . The two lines at which errors are returned correspond to scenarios that should not occur if the conditions of Theorem 7 hold. In line 14, we visit a non-input node at time . But such a node should never have been reached in the recursion, since we only followed edges that have -information flow. Its presence, therefore, would contradict the computational system model. In line 5, is marked “invalid”, implying that there is no path leading to it from the input nodes. Once again, this can only occur if the computational system model is violated, or if the conditions of Theorem 7 do not hold.
On Computational Complexity
The complexity of this algorithm is exactly that of Depth-first Search, [56, Sec. 22.3]. To be precise, we consider the computational system to extend until the time of the output node, i.e., we take . So the complexity of the algorithm is . This is easily verified: if we assume that all edges in the system have -information flow, then all edges and nodes must be traversed by the search. At each node, we must execute lines 7 through 14, and 26 through 28, which take a constant amount of time. Since we have nodes over time points, this adds up to steps. We also need to execute the loop in lines 15 through 24, which counts the number of incoming edges at every node. For all nodes combined, this adds up to steps.
If the graph is fully connected as described in Section 2, then and , so the effective complexity is just . However, if we know that the underlying graph is sparse (e.g., because of anatomical priors in neuroscience), then we may have , or even , bringing down the complexity of the search. It should be noted that in either case, the complexity of identifying which edges have -information flow is potentially exponential in , as discussed later in Section 7.2. This is much larger than the complexity of tracing out information paths, so finding edges with -information flow is, in fact, the “hard part” of the problem.
5.4 Derived Information and Redundancy
The framework we develop for information flow allows one to obtain a more fine-grained understanding of information structure in a computational system, especially when compared with classical tools such as correlation and phase synchrony [68, 69]. This allows the experimentalist to better investigate the nature of the computation being performed. A concept that we believe will be extremely useful in this regard is one we call “derived information”, which is defined below.
Definition 12** (Derived -Information).**
In a computational system , a transmission is said to be derived -information of a different transmission if —— forms a Markov chain. That is, the following condition must hold:
[TABLE]
So, adds no new information about , when given . The same definition extends to transmissions on sets of edges. Note that, as far as the definition is concerned, and may be any two arbitrary points in time. However, we will typically consider cases when .
One potential use-case scenario for derived information arises in the context of redundant flows. Consider the computational system presented in Figure 4, originally described under Counterexample 3. We see two edges sending the same transmission to the node . This is an example of what we call “redundant transmissions”. In general, since we only consider information about to be relevant, the exact transmissions communicated over two edges at a given point in time may be different. But if they convey the same information about to a given node, then we view them as essentially redundant. Definition 4, when applied to this system, will detect both these edges as having -information flow, since given , their transmissions depend on . In the notation of the Separability property mentioned earlier (Proposition 9), both edges as well as will belong in the set .
Derived information provides a general methodology to understand when transmissions on certain edges may be redundant. Naturally, if the transmissions on two edges and are redundant, then they must be derived -information of one another. This amounts to checking two more conditional independence relationships, for which consistent tests exist in the limit of infinite trials, as discussed in Section 5.2.
In the following section, we shall see another application of derived information; when applied to specific sets, it can in some cases be used to detect the presence of hidden (unobserved) nodes. Later, in Section 6.3, we discuss an example where the notion of derived information helps us make a new kind of inference about the fine structure of information flow, one that would not be possible using tools such as Granger Causality and Directed Information.
5.5 Hidden Nodes
In Section 5.3, we showed how the Information Path Algorithm may fail to discover -information paths if one of the assumptions of the computational system model or the observation model breaks in some way. Here, we discuss one specific situation in which the observation model may break, i.e., when not all nodes are observed. We call these unobserved nodes “hidden nodes”, and assume that we do not see transmissions on incoming or outgoing edges of these nodes.
Definition 13** (Hidden nodes).**
Consider a computational system defined on the time-unrolled graph as before. Suppose that only a subset of nodes in this graph are observed. Specifically, if was the original set of nodes in , prior to time-unrolling, then we observe only the nodes , where is a set of unobserved nodes called hidden nodes.
To describe the observed component of the computational system, we define , and . Also let . Finally, we set up the observed component of the computational system as before: . Thus, we only observe the transmissions on edges in . As usual, we denote the set of all hidden nodes at time by , and the set of all observed nodes at time by .
The presence of hidden nodes of this nature implies that much of the theory we have developed will not apply. Lemma 4 no longer truly holds, in that information about may persist in the system by passing through the hidden node, even if no observed edge has -information flow. So, naturally, Property 1 also fails to hold. Hence, we are not guaranteed to be able to identify all edges with -information flow, and discover all -information paths as before. For example, refer to the cases shown in Figure 6, where we no can longer find -information paths because of the presence of a hidden node.
Fortunately, at least in some cases, the concept of derived information (Definition 12) provides a simple way to tell whether or not a hidden node exists. Specifically, if at some time , a hidden node transmits information about which is unavailable within the system at that time, and which is utilized by some node at the next time instant, then the set of all observed transmissions will not be derived -information of the set of all transmissions at time . In other words, the Global Markovity condition (Corollary 5) on the observed graph, ——, will break. Unfortunately, the notion of “utilization” is difficult to express mathematically, without resorting to the use of ideas from causality that are based on intervention. The result we prove, therefore, is a simpler sufficiency argument, which guarantees the presence of a hidden node if the aforementioned Markov condition is observed to break. This result is proved in Proposition 11, but first, we define some adjectives.
Definition 14** (-relevant hidden node).**
A hidden node is said to be -relevant if carries -information flow in . Similarly, a subset of hidden nodes is said to be -relevant if carries -information flow in .
Definition 15** (-derived hidden node).**
A hidden node is said to be -derived if the Markov chain —— holds. Similarly, a subset of hidden nodes is said to be -derived if the Markov chain —— holds.
Lemma 10**.**
If a subset of hidden nodes is not -derived, then it is -relevant.252525If this lemma appears to be somewhat strong, it is only because of the nomenclature “-derived”. For our purposes, a hidden node whose transmissions are independent of the message is also -derived, since it satisfies the aforementioned Markov condition.
Proposition 11**.**
In a computational system with hidden nodes, if Global Markovity on the observed graph, , fails to hold from time to , i.e. if I\bigl{(}M;X(\widetilde{\mathcal{E}}_{t+1})\,\big{|}\,X(\widetilde{\mathcal{E}}_{t})\bigr{)}>0, then the hidden nodes at time are not -derived.
Proofs of Lemma 10 and Proposition 11 are very straightforward, and are provided in Appendix D. As a direct consequence of these two results, if Global Markovity fails to hold on the observed nodes from time to , then is -relevant. By Proposition 1, this simply means that there exists at least one -relevant hidden node at time .
Although Proposition 11 appears to provide a straightforward mechanism to test whether or not hidden nodes exist, it does not always work. If a hidden node’s transmissions have no -information flow, then the node will not be detected. But in this case, it could be argued that such a hidden node does not change whether information paths can be identified, and so can be subsumed by one or more of the intrinsic random variables . Such a hidden node is, therefore, classified by Definition 14 as not -relevant.
However, to make matters worse, the converse of Proposition 11 does not hold. In particular, there may exist an -relevant hidden node at time , whose transmission is ignored by the node that received it, so that the Markov chain —— continues to hold (see Figure 7). Such a hidden node may still be considered largely innocuous.
The most serious case of a hidden node going undetected is one that contains an -derived hidden node, whose transmission is used by the receiving node while performing its computation; however the hidden node’s transmission is “masked” by a redundant transmission from an observed node (see Figure 8). In this case, Global Markovity on will not break, yet the hidden node’s transmission may be instrumental in producing a certain output distribution. In some instances, such hidden nodes can be detected by checking for Local Markovity (Proposition 6; see Figure 8(a)). However, there are still cases where if we were somehow able to intervene and delete the transmission of the hidden node, then the computational system’s output may not remain the same, despite the existence of a redundant transmission from an observed node (see Figure 8(b)). Indeed, the presence of redundancy in such a scenario does not guarantee that the computational system will actually leverage it.
5.6 On Multiple Messages and the Distribution of the Message
Just as we can infer information flow and information paths for a single message, we can examine the flows of multiple messages in the same computational system. Consider a case where we wish to understand the information flows of two messages, and . An neuroscientific example of this might be information flow about two independent components of a visual stimulus, e.g., shape and color (such as in [2]). If M_{1}\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}M_{2}, then we could separately identify edges and paths that have -information flow and -information flow, by applying the theory and algorithm as-is for each message individually.
However, if the two messages are dependent on one another, one could end up confounding their information flows, based on how they depend on each other, and how the computational system’s transmissions carry their joint information. As a simple example, consider the system shown in Figure 9, where and , with . Clearly, and both share some redundant information in , and bit. Thus, we will see -information flow as well as -information flow on both edges, since the transmission of each edge satisfies I\bigl{(}M_{i};X(E)\bigr{)}>0 for .
Consider what this means for the aforementioned example of shape and color of a visual stimulus. If a neuroscientist expects that the information paths corresponding to shape and color in the brain are different from each other, what is the best way to design stimuli so as to bring out this difference? Suppose they decided to present a total of four different stimuli, , with two different shapes and two different colors. Let be the first bit of the binary representation of , denoting shape, and be the second bit, denoting color. Now if the neuroscientist chose to present stimuli with a uniform distribution over , i.e., if each shape-color combination was shown for one-quarter of all trials, then and would be independent of each other, and their individual flows could be tracked separately. However, if the neuroscientist chose to present the four possible stimuli with probabilities respectively, then and are no longer independent of each other, and it may become hard to separate their individual flows as in the example in Figure 9.
These examples suggest that, when trying to understand the flows of different messages in a computational system, it helps if they are independent of one another. So from the perspective of experiment design in a neuroscientific context, it is often more sensible to design stimuli so that the two messages of interest are independent of one another. Even when considering a single message that takes one of several values, it becomes important to appropriately choose a distribution over these values to ensure that any sub-messages that are of interest remain independent of one another. This would allow the experimentalist to better understand how “independent dimensions” of the stimulus are processed in the brain.
However, there are also situations where the experimental paradigm necessitates a statistical distribution of stimuli that makes two sub-messages of interest dependent on one another. For instance, the Posner experimental paradigm for attention [70] only works when the proportion of “valid” trials (a certain type of trial specific to this paradigm) is roughly 70%. Similarly, during data preprocessing, it is common to discard trials that are excessively noisy, based on some predetermined metric: this process could skew the distribution of the message, even if the original distribution was uniform. If it is still of interest to understand the individual flows of sub-messages in this case, then a possible solution might then be to sub-select experimental trials in such a way as to keep the two sub-messages independent of one another.
6 Canonical Computational Examples
In this section, we provide a few canonical examples for computational systems from various contexts. In each case, we discuss what the message is, and identify which edges carry -information flow. We also explain how the path recovered by the information path algorithm might be the intuitive choice in each example.
6.1 The Butterfly Network from Network Coding
For our first example, we cover the butterfly network from network coding literature [12, Fig. 7b], reproduced here in Figure 10. We consider two different messages, , provided as input to the system. Edges along which information about flows are colored in blue, while edges along which information about flows are colored in orange. The reader may identify these using Definition 4 and the transmission on each edge shown in Figure 10.
An important feature to observe is that when mixes information by computing the xor of and , we see information about spontaneously beginning to flow on and similarly, information about beginning to flow on . This is expected, since is relevant for decoding at this stage, and indeed, it is exactly this idea which is used to decode at . All of this is true, despite the fact that is independent of and individually. This is once again, a prime example of synergy in action.
Applying the information path algorithm for the message at will reveal two paths: the “upper path” , and the “middle path” . However, applying the information path algorithm for the message at reveals that exclusively used the “middle path”, , to arrive at from the input nodes.
6.2 The Fast Fourier Transform
The Fast Fourier Transform (FFT) is a well-known computational network that provides an intuitive setting for examining information flow. In general, the -point FFT is an implementation of the -point Discrete Fourier Transform (DFT), given by
[TABLE]
The DFT is a basis transformation of a discrete-time signal , which is usually assumed to be periodic with period . The -point DFT represents such a signal in the complex-exponential Fourier basis, yielding the Fourier coefficients . We consider a simple 4-point DFT, i.e. . The FFT implements this transform using the computational system shown in Figure 11. We refer the reader to [44, Ch. 9] for details. For notational convenience, we have set .
We use this example to demonstrate how the definition of the message is important in determining information flow. First, suppose the message is one of two signals: , or . This can be written as and , where is the Kronecker Delta function, and we assume . The full computational system, along with the random variables computed on all edges, is shown in Figure 12. The edges that have -information flow are highlighted in blue. Since is encoded into the even part of (observe that ), we notice that only the “even component” of the FFT system (corresponding to the 2-point FFT on the even indices of ) is active [44, Sec. 9.3]. Furthermore, only , the DC component, and , the first harmonic, show variation with at the output, as we would expect based on the two input signals.
As a second example, consider the case shown in Figure 13. Here, the message is again one of two signals: , or . These signals can be jointly expressed in terms of the binary message random variable as . The two signals are flat in their magnitude spectra and differ only in their phase, creating -functions in the Fourier domain that are frequency-shifted with respect to one another: . Once again, the edges in the network that carry -information flow are demarcated in blue. A detailed derivation of the values of the transmissions in the computational system can be found in Appendix F.1.
These two examples make it clear that, based on how the message is defined, the -information paths in the system can be very different. Indeed, if the message were as general as possible, by placing a probability distribution over all possible values of in , we know that all edges in the computational system would have -information flow. However, selectively restricting to just a few signals helps reveal some kind of structure within the FFT network.
Another feature that can be observed in these examples is how the output of the computational system can be a function of the message. Although only very simple functions of the message have been shown at the outputs here, the FFT demonstrates that, in principle, more complex functions of the message may also be generated.
6.3 The Schalkwijk and Kailath Scheme
The Schalkwijk and Kailath scheme [71] is an efficient strategy for communicating a message in the presence of a noisy feedforward channel and a noiseless feedback channel. We have previously used this scheme as a counterexample [41], to show that comparing Granger causal influences in forward and backward directions can lead to erroneous inferences on the direction in which the message is being sent in this feedback system. We first provide a brief overview of the scheme, then recapitulate our previous result, and finally demonstrate what the information flow framework developed in this paper has to offer in the case of this example.
Consider the communication system depicted in Figure 14, which shows the schematic of a simplified version of the Schalkwijk and Kailath scheme. For convenience, let us denote the transmitter, , and receiver, , by Alice and Bob respectively. Alice is attempting to communicate a message to Bob over an additive Gaussian channel, but in the presence of noiseless feedback. Alice starts by transmitting the message to Bob, over the noisy feedforward channel. Bob receives a corrupted version of , given by , and computes an estimate . He sends this estimate back to Alice over the noiseless feedback channel. In the iterations that follow, Alice computes the error in Bob’s most recent estimate, , and sends this to Bob over the noisy feedforward channel. Meanwhile, Bob updates his estimate based on Alice’s noisy transmissions , using the following rule:
[TABLE]
Thus, this strategy ensures that Bob’s estimate converges to in mean squared sense [41].
Intuitively, one might expect that, since the message is being transmitted in the forward direction, the Granger Causal influence from Alice to Bob is greater than that from Bob to Alice. However, our earlier result [41] showed that, in fact, the opposite is true. In other words, even though the message is being communicated from Alice to Bob, the Granger Causal influence from Bob to Alice is greater; in fact, the Granger Causal index from Bob to Alice is infinite. The reason for this is that, while Alice’s past transmissions do not perfectly predict Bob’s transmissions (due to the presence of noise in the feedforward link), Bob’s past transmissions perfectly predict Alice’s transmissions (since the latter are a simple function of the former). Therefore, the Granger Causal index from Alice to Bob, which measures the relative predictive gain of including Alice’s past transmissions in the autoregression for Bob’s transmissions, remains finite; while the Granger Causal index from Bob to Alice becomes infinite.
Our earlier paper on this subject [41] concluded that the direction of greater Granger Causal influence could be opposite to the “direction of information flow” in the Schalkwijk and Kailath scheme. There, “information flow” was being used purely in an intuitive sense, to mean the direction in which the message was being communicated in that system. The intent of our previous paper was to explain that it is not always possible to interpret a larger Granger causal influence in a certain direction to mean that a specific message is being communicated in that direction. In contrast, this paper presents a refined theoretical framework that defines information flow about a message for a specific edge in a computational system. Now, we no longer speak of one specific direction in which information flows; rather, we describe which edges carry information about the message in their transmissions at each point in time. This leads to a more nuanced understanding of information flow in the Schalkwijk-Kailath setting.
Before we can analyze the -information flows in the Schalkwijk-Kailath scheme, we need to fit the scheme within the computational system framework. Figure 15 shows the time-unrolled computational system corresponding to two feedforward and feedback iterations of the simplified Schalkwijk-Kailath scheme described before. In order to translate the communication system into our computational system model while remaining consistent with our earlier work [41], we have merged the process of noise addition with the receiver, i.e., Bob. This exposes the edges with Alice’s and Bob’s transmissions, making them observable, as was assumed in our previous paper [41]. This is also consistent with what would have been observable if and were neurons (or neural populations) whose outputs a neuroscientist were to measure.262626From a wireless communication system perspective, as well, it is more reasonable to assume noise to be a part of the receiver’s node, since the additive noise in a signal is usually considered to be the result of thermal noise in the receiver’s circuitry. Note that one full iteration of the Schalkwijk-Kailath scheme takes two time steps in this model, so the iteration index advances once for every two time steps . Also, note that this does not make or “hidden nodes”, since the function computed at can be defined purely in terms of its inputs, , and its intrinsic random variable, (which absorbs ), as follows:
[TABLE]
where takes the role of the noise in the communication system. Also, to understand the time index for node , note that in the first step of iteration , Alice transmits to Bob, i.e., node transmits to (see Figure 15).
Now, we first show that all edges depicted in blue in Figure 15 carry -information flow, based on Definition 4. Specifically, both Alice’s feedforward transmissions and Bob’s feedback transmissions have -information flow. This should not be surprising for the following intuitive reasons: Alice’s transmissions convey information about which Bob uses to improve his estimate; meanwhile, Bob’s transmissions are estimates of , and therefore must depend on . In fact, we can take this intuitive argument further: suppose we were to quantify -information flow by using the following natural extension of our definition,
[TABLE]
Noting that Definition 4 only specified whether or not a given edge had information flow, all that we have now done is to take the maximum over the subsets of edges used to discover -information flow in that definition. This quantification is fully consistent with our definition of -information flow, since it goes to zero if and only if the -information flow on an edge goes to zero. Now, using this quantitative notion of information flow, we can ask how the -information flow on a given link—feedforward or feedback—varies with time. In particular, it should be intuitively clear that the -information content in Bob’s transmissions, i.e. , increases over time as his estimate improves. This is depicted as an increase in the thickness of the edges carrying Bob’s transmissions with time. Meanwhile, the information content in Alice’s transmissions decreases with time. To understand why the latter is true, note that, after the first iteration, Alice’s transmission represents the noise in Bob’s estimate. Therefore, just as in Counterexample 1, when conditioned on Bob’s estimate, Alice’s transmissions depend on the message. At the initial iterations, when Bob’s estimate is poor, we must have that is very small. Hence, we see that:
[TABLE]
where in (a), the second term is zero because is a constant when given and ; while in (b), is assumed to be approximately zero when Bob’s estimate is poor (as we might expect if the noise is large, for instance). Hence, for the first few iterations, the quantified -information flow of Alice’s transmissions is close to , which is as large as the flow can get. However, as Bob’s estimate improves, becomes closer to , and therefore I(M;Y_{i}\,\big{|}\,\widehat{M}_{i-1}) becomes close to zero. At the same time, is equal to zero, since carries only information about the noise in (after the first iteration), which is independent of . Thus, the quantified -information flow of Alice’s transmissions decreases over time. Correspondingly, this is depicted using edges whose thickness decreases over time in Figure 15.
Quantifying the -information flows of the feedforward and feedback links thus reveals an asymmetry between Alice and Bob that strongly suggests that the message is being transmitted from Alice to Bob. However, we can get a more nuanced understanding of information flow in this system by asking whether Bob’s transmissions are derived from Alice’s, or vice versa. First, consider whether Bob’s transmissions are derived -information of Alice’s previous transmissions: this can be expressed in terms of the Markov chain ——. Observe that this Markov chain holds trivially:
[TABLE]
However, if we consider whether Alice’s transmissions are derived -information of Bob’s past transmissions, it can be shown that —— is not a valid Markov chain (see Appendix F.2 for a detailed derivation). Hence, we see that Bob’s transmissions are derived -information of all of Alice’s past transmissions, however, Alice’s transmissions are not derived -information of all of Bob’s past transmissions. In conjunction with the fact that the volume of -information flow in Alice’s transmissions slowly decreases from with time, while the volume of -information flow in Bob’s transmissions slowly increases to with time, this suggests that Alice has some information about the message that Bob slowly receives from Alice.
This example shows how a measure that quantifies information flow, along with derived information, can be used to understand some finer computational structure present within the computational system. In general, however, care needs to be exercised in applying derived -information: one must choose what Markov condition to check in a principled manner. In the specific case of the Schalkwijk-Kailath example, we had the advantage of being in a two-node setting, where the derived information expressions we examined had clear interpretations. It may be that analyzing information flow first, to understand which variables transmit information about to one another, can help guide the choice of variables to examine when applying derived -information.
6.4 A Message Defined at the Output of a System
We now describe an example where the message is defined at the output of a computational system, instead of at the input. Although Definition 3c defines the message to be a random variable available at the input nodes, it is also possible to define the message at the output of the computational system. In this scenario, the input nodes are no longer well-defined as per Definition 3c. Instead, we would define output nodes in the same manner.272727Note, however, that the corresponding “opposite” of Theorem 7 (wherein the places of “input” and “output” nodes are switched) does not hold in this case. That is, it is not true that if at some previous time instant, an “input” node’s outgoing transmissions depend on the message, then there is an information path connecting that input node to the aforementioned output nodes. The reason this fails is that there could be a “source” node at an even earlier time instant, which provides information about to both the input node under consideration, and the output nodes, via two separate, diverging paths. Therefore, there may be no path from the input node to the output nodes.
Consider the computational system shown in Figure 16. The system on the right executes the function depicted by the boolean circuit shown on the left. is an external parameter, which is taken to be a fixed constant. When , the and gate at the top is activated while the and gate at the bottom is deactivated, so the message depends only on . In this case, only the edges shown in blue have -information flow. On the other hand, when , the opposite happens, and the message depends only on . Now, only edges shown in orange have -information flow. If was not a deterministic external parameter, but a random variable itself, then all edges shown in the figure would have -information flow, since would depend on all their values.
So, we see that when the message is defined at the output, the “origin” of the message may be from within the computation system itself, in the form of one or more intrinsically generated random variables: here, either or . The notion of information flow and information paths can thus help us identify where the message originates within the computational system.
Furthermore, just as information paths can change depending upon how the message is defined (as in Section 6.2), information paths may also change depending on external parameters: inputs such as that are fed into the computational system, which are not part of the message. These inputs essentially shape the nature of the computation being performed, and so, naturally, they can affect information paths.
7 Discussion
This paper presented a theoretical framework for defining and studying information flow of a specific message in a computational system. The core contribution of our paper was a definition for information flow that is concretely grounded in the computational task and intimately tied to a specific message. This relied on another important contribution: the development of an underlying computational model, which enables the interpretation of statistical analyses. After providing a clearly-defined model for a computational system, we presented several candidate definitions for information flow along with counterexamples and showed that our definition, which is based on positivity of a conditional mutual information expression, satisfies several intuitive properties, whereas other candidate definitions do not. We then examined these properties in detail and showed, in particular, that our definition naturally leads to the existence of “information paths”. We also discussed how information flow can be inferred through conditional independence testing, and provided an algorithm for recovering the information paths in a given system. Finally, we studied some canonical examples of computational systems from different contexts, and showed that our definition of information flow is intuitive in each case.
We proceed to discuss several important assumptions and simplifications in our model. We also discuss existing literature related to estimation of causal influence in neuroscience, and how our computational system model leads to a significantly different measure of information flow. Similarly, we discuss how our framework is very different from the field of Probabilistic Graphical Models.
7.1 Neuroscientific Concerns
7.1.1 Observing edges vs. nodes
The observation model stated in Section 5.1 makes a crucial assumption, namely, that transmissions on each edge can be observed. In neuroscientific experiments, however, we often record activity from single neurons (as in the case of electrophysiological recordings), or aggregate activity from groups of neurons (as with Local Field Potentials measured in Electrocorticography and Electroencephalography). These neurons, or groups of neurons, are considered to be nodes communicating to one another in a network. It may not be known which nodes are connected to which other nodes, let alone the recipient of each transmission at every time instant. This is a marked departure from our assumption that transmissions on edges can be observed. To some extent, it is possible to incorporate a “node-centric” model within our computational system by assuming that all nodes broadcast their transmissions. However, that still leaves unanswered the question of which nodes actually “hear” another’s transmissions. A possible resolution to that question might arise from an understanding of receiver response. That is, we consider a revised model in which an edge exists if a receiving neuron uses the information transmitted by some neuron at the previous time instant. This issue is beyond the scope of the current work, and will be addressed in subsequent studies.
We also note that, although tools based on Granger causality implicitly assume that nodes are measured and not edges, they do not resolve the issue of which node is “talking” to which other node. For example, if two different nodes and communicate the same information to a third node, , any regression based analysis will assign a weight of one-half to each of and . However, the true function, , may be using only the information coming from , or only the information coming from , or using the two in some other unequal proportion. Such cases may only be identifiable through an interventional approach.
7.1.2 Observing memories
Another important assumption in the observation model is that memories of nodes are observed as transmissions on self-edges. If these transmissions are implemented in the form of some internal state at each node, then they might be difficult to observe in practice.282828If every node represents a group of neurons, however it may just be that their internal state is represented in the form of communication between these neurons. In that case, perhaps observing their internal state is just a matter of having more spatially refined measurements. It remains to be fully understood whether one can compensate for not observing memories in some manner, e.g., by assuming that the memory of a node is the full history of its transmissions and receptions. While this means that intrinsically generated random variables that are not propagated to other nodes will never be observed, it could be argued that such variables could have no impact on the system (save for acting as “computational noise”). So perhaps it suffices to observe only transmissions between different nodes (and not self-edges). Further work is required to understand what ramifications such an assumption has on identifying information flows and information paths.
Conversely, our work may suggest to neuroscientists that inferences about information flow are more reliably obtained if one can measure transmissions on edges in the graph, rather than transmissions of nodes. This may call for newer imaging modalities, or new uses of existing modalities, such as treating axons as targets for invasive recordings, perhaps at nodes of Ranvier. Further, perhaps if one wishes to observe memories, it is important to measure not only spikes, but also membrane voltages (e.g. using voltage-sensitive dyes [72] or, less directly, through measurements of changes in neurotransmitter concentrations outside a cell [73]).
7.1.3 Discretization of time
Yet another implicit assumption in our computational system model is that transmissions occur at discrete points in time. This assumption is justified for synchronous digital circuits used commonly today, or if the computational system of interest is a trained artificial neural network, for instance. However, this is not a perfect model of the brain, because neural spiking (among other processes), does not occur only at multiples of some fundamental unit of time. This issue might be partially mitigated by assuming that neural computation happens at a certain time scale, and by using a sufficiently high sampling rate so that Nyquist-rate-type arguments apply. This may not be possible in certain modalities (e.g. Calcium imaging and functional Magnetic Resonance Imaging) that are inherently slow, however, so it would be interesting to understand what inferences we are no longer capable of making. Alternatively, if the sampling rate is too high, it may be useful to consider windows within which to look for -information flow. The exact implications of using such preprocessing methods will also need to be studied in greater detail, in future.
7.1.4 Message enters at
Another assumption in our framework is that the message enters the system at, and only at, time . This is essential, given the way we have defined input nodes: nodes at time , whose outputs depend on the message (and which have no other shared source of randomness). However, this assumption does not allow for a dynamically evolving stimulus, which is also common in neuroscientific experiments. Suppose we allow the message to enter the system at a later time instant, say at some node , for , i.e., may compute a function not just of its inputs, but also of . Then, if we want the information path theorem to continue to hold, we must also add to the set of input nodes.292929We should also expect that any Local Markovity conditions at time (see Proposition 6) that involve the node will no longer hold. Thus, if we see dependence at some other node , at a later time instant , the information paths leading to may arise from the original input nodes or from , or both. As we might intuitively expect, the more time points we allow the message to enter at, the more such information paths we will likely see, making the results of our analysis harder to interpret.
7.1.5 Experimental design and the message
An important aspect of our work is that it explicitly incorporates the message, which in neuroscientific experiments is often some information contained in the stimulus. This aids the neuroscientist in designing experiments, for example, in understanding what stimuli will help them make a certain inference about information flow. In particular, one needs to use at least two different stimuli in order to obtain any determination about information flow. While this is implicitly understood in neuroscience, as evidenced by comparisons with baselines, or by the use of permutation tests to scramble stimulus-trial correlations for a null model, our framework provides a more direct method for identifying and interpreting stimulus-related information flow.
7.2 The Difficulty of Estimation
A strategy for detecting edges that have -information flow was presented in Section 5.2. In practice, however, there are several issues associated with employing such a strategy. These are discussed below.
Firstly, we currently assume that observations are noiseless (see Section 5.1, Assumption 3). It is unclear, exactly, to what extent noisy observations will impact the inference of information flow. In particular, it is worth understanding whether small amounts of observation noise can be tolerated if all edges with -information flow have a sufficiently large “volume” of information (i.e., the corresponding mutual or conditional mutual information is sufficiently large). As was described intuitively in Section 5.2, if the information volume is large, then even under noisy conditions, we might expect the test statistic to clear the threshold, so the presence of -information flow can still be detected consistently. But small volumes of information that aggregate over time—e.g. information about “trickling” over time from one node to another—could still pose issues. Such -information flow could go undetected, as has been shown to occur in different contexts [38], using different measures of flow. It is possible that Derived Information, in particular, is hard to infer in the presence of noise. This could make the task of detecting the presence of a hidden node difficult (consider the case of a “trickling” hidden node), as well as that of identifying redundant links.
Secondly, detecting whether each edge at time has information flow involves checking all subsets of . For nodes and edges, this implies subsets of edges that need to be searched. This could be seen as being prohibitively difficult for , or for greater than about 5 or 6 nodes. However, in reality, graphs in neuroscience are often known to be edge-sparse [74, 75, 76]. For example, in the brain, a well-established 11-node network is the reward network [76]. Most nodes in this network typically have just one incoming and one outgoing connection. The two most important nodes have five incoming edges each, with two and four outgoing edges respectively. Further, it is known which connections are inhibitory and which are excitatory, which could further help with testing for information flow. A fully connected network would have had 121 edges, but the underlying connectivity of the circuit only allows for a total of 17 edges in this network. So in reality, anatomical priors help reduce the number of edges to well within the range of what is computable. Nevertheless, it remains of interest to find methods by which nodes and/or edges can be excluded from the search, and this could be another topic for further research.
Another statistical issue that crops up when attempting to simultaneously perform several conditional independence tests is the problem of multiple comparisons [77]. Simply put, when performing a large number of independent hypotheses tests, say , at some fixed false alarm rate , on average, we should expect of these tests to erroneously reject the null. In the context of information flow, we might wish to set the null hypothesis to be the absence of -information flow on a given edge. Then, to test for -information flow on this edge, we need to perform a large number of conditional independence tests—call this number —at some false alarm rate . These tests are, in fact, not independent of one another; nevertheless, very loosely put, if we choose a false alarm rate , we may find that the probability of at least one false alarm is too high. This would make us erroneously infer that this particular edge has -information flow; moreover, since this argument applies to any edge, if is not chosen conservatively enough, we may erroneously infer that all edges have -information flow. This multiple hypothesis testing problem is better posed as a “Global Null test” (e.g., see [78]), wherein the global null is the hypothesis that all of the conditional independence tests are individually null (i.e., that there is no -information flow on the given edge), and the global alternative is the hypothesis that at least one of the conditional independence tests is non-null (i.e., that there is -information flow on the given edge). As mentioned before, however, the conditional independence tests dictated by Definition 4 are, in general, dependent on one another. Furthermore, it might not be easy to describe the manner of dependence, so when choosing methods that control the family-wise error rate, it is essential to choose those that work under arbitrary dependence. A simple example of such a test is the well-known Bonferroni correction, which uses a level for each test (where is the desired false alarm rate for the overall global null test); but we may find that such methods have insufficient statistical power. A potential solution to this problem might involve combining multiple global null tests in some meaningful way: for example, one could imagine designing a procedure that controls the False Discovery Rate303030These methods control the expected proportion of false discoveries, i.e., the proportion of null hypotheses that are falsely rejected. [79] on the identification of edges with -information flow.313131Care is needed when doing this, however, since tests for -information flow on different edges at the same time instant are also dependent on one another. Another approach might be to find ways of directly testing information paths, wherein the hypothesis tested would be that a certain -information path exists in the system, rather than requiring every edge with -information flow be identified first. All of these ideas are potential avenues for future work.
7.3 The Limitations of Granger Causality and Related Tools
Mapping directed functional connectivity and information flow in the brain has been a hot topic for several years, as evidenced by the large body of work in this direction [20, 21, 22]. Approaches for statistically mapping functional connectivity often rely on variations of Granger Causality [24] and, more recently, Directed Information [26, 27, 28], which we here collectively refer to as “Granger Causality-based tools”. These approaches lack a systematic framework that ties the statistical analysis to the underlying computation, however, and the interpretations drawn from their use have often been questioned [31, 33, 34, 41, 38, 39, 40].
In particular, a crucial difference between our approach and that of Granger Causality-based tools is that the latter do not have an explicit description of the message. Instead, they provide mechanisms to condense a pair of time series into a single statistic. There are no concrete models that can be used to interpret what this statistic means for the flow of information about the message. Furthermore, if one is interested in the information flow of multiple messages, Granger Causality-based tools do not provide an immediate solution. This is why a tool that ties information flow directly with a message is of great interest to practitioners.
The absence of an underlying computational framework with well-defined assumptions inherently makes it very hard to draw sound inferences through the application of Granger Causality-based tools. A striking example of this is a recent result of ours [41] that shows, using a feedback communication system, that the direction of greater Granger-causal influence can be opposite to the direction in which the message is communicated, even in the absence of hidden nodes and measurement noise. The time-unrolled graph framework presented here has been specifically designed to address this issue, and present a clear understanding of information flow, even in the presence of feedback. The example given in Section 6.3 demonstrates a potential resolution to this issue.
Granger Causality was originally developed for the study of time-series that occur only once, such as in economics [23]. An artifact of this development is that it was not designed to incorporate multiple trials of the same process. Instead, it assumes stationarity to help estimate parameters of the random variables that control the process. In the neuroscientific context, stationarity is often a very poor assumption, since the segment of time-series data corresponding to each trial may be short, and often sees some kind of stimulus presentation. Naturally, presentation of the stimulus changes the underlying parameters of the time-series and destroys stationarity; indeed, this is the quintessential aspect of the experiment. Thus, in order to understand processing in such stimulus-driven tasks, one needs to be able to infer time-dependent information flows from data. While information-theoretic extensions of Granger Causality such as Transfer Entropy and Directed Information do not assume stationarity, they nevertheless fail to provide a dynamically evolving picture of information flow.
Lastly, it is unclear whether the directional influences estimated using Granger Causality-based tools have any correspondence with the rigorous notion of information flow we have derived here, under special assumptions, e.g., Gaussianity. This is a promising future direction as well, since it is important to understand in which situations these methods recover meaningful flows of information, and in which cases we must be careful with interpretation.
7.4 Probabilistic Graphical Models and Pearl’s Causality
There is one important difference that distinguishes our work from the perspective adopted in the field of probabilistic graphical models (PGMs) [80], and the representations therein. In our framework, nodes represent computational units, whereas in PGMs, nodes represent the random variables themselves, and edges capture the conditional independence relationships between these variables. While it might be possible to construct a PGM that is equivalent to our computational model, this would likely eliminate any intuitive structure captured by the computational graph.
It remains to be understood whether and how Pearl’s notions of causality [13] can be seamlessly merged with the understanding of information flow developed here. We expect that some formal application of causality will be needed in going from an edge-centric model (as presented here) to a more node-centric one (discussed in Section 7.1), in order to identify which transmissions influenced a given node’s output.
There are several works in the literature that discuss measures of information flow in probabilistic graphical models [81, 82], but they are heavily inspired by causality and largely center around an interventionist approach. In contrast, our definition of information flow is based on a computational system model that translates more readily to neuroscience, and we assume that the experimentalist is restricted to making observations.
7.5 Future Directions for Theoretical Development
A natural question that arises from this paper is: how can our definition of information flow on an edge be extended to a more generic information measure, which also quantifies the volume of flow? Finding such a measure will involve aggregating the conditional mutual information for each subset of edges into a single value (one example of such a measure was provided in Section 6.3, though it was not developed from first-principles). It is as yet unclear how this might be achieved, while still gelling well with our intuition of what this information flow volume ought to be. We believe that the right approach is to start by designating a set of properties that we would like information flow volumes to satisfy, and then to propose a measure through the use of representative examples and counterexamples.
A second direction that emerges is related to Partial Information Decomposition (PID) [47, 48, 49], which was discussed earlier in Section 3.5. -information flow is very closely related to the PID: while Candidate Definition 1 checks for positivity of mutual information between and , and hence implying the presence of unique and/or redundant information, our definition also detects the presence of purely synergistic information. Since our definition is closely tied to computation and is strongly motivated through the goal of finding unbroken information paths, the close relationship between PID and our definition suggests that PID might be the right toolset for obtaining a more fine-grained understanding of information flow, as well as computation. In particular, it would be useful to know how the understanding of computation is enhanced through a PID analysis, which describes the unique, redundant and synergistic components of the message in different nodes’ transmissions. Finally, we note that the PID could also help inform the discussion on a definition for information volume. Providing a useful definition of information volume based on current definitions of unique, redundant and synergistic information, and asking whether the problem of information flow can inform the PID literature, will also be the subject of future research.
A third direction has to do with alternate definitions of information flow: the properties we stated in this paper are not sufficient to uniquely specify our definition of information flow. For example, the all-zero function as well as the all-ones function satisfy the Broken Telephone property, although they are not particularly useful definitions of information flow. Thus, it would be useful to understand what other properties we should impose so as to arrive at a unique definition of information flow. As a crude and preliminary example, we demonstrate how this might be done in Appendix E.
7.6 Concluding Remarks
We conclude by describing some of our general impressions in working on the theoretical development presented in this paper. As such, these points merely highlight some of our opinions on how theory—and more specifically, information theory—may be applied in neuroscience.
As mentioned in the introduction, we drew inspiration from two papers that discuss how experimentalists understand systems in biology and neuroscience [15, 16]. Both these works advocate for theory by arguing that we need new analytical tools, and that the accumulation of empirical knowledge alone does not constitute understanding. Lazebnik [15], in particular, mentions how terminology in biology tends to be vague and non-committal. We feel that an important reason for this is the absence of concrete underlying models, with clearly-stated assumptions. In other words, we think that theory and modeling can go a long way in providing a language that will enable well-grounded discussions. This language, in turn, arises through the development of theoretical models and formal definitions.
Another point made by both the aforementioned papers is that we should attempt to understand large computational systems by first examining smaller models, and models in which the ground truth is already known. This approach allows us to create new analytical tools that can be thoroughly vetted, so that the interpretations drawn from their use in experimental practice is unambiguous and undebated. We also believe that when trying to understand large computational systems, it is essential to start with toy models such as Counterexample 1. This philosophy of starting with toy models, and abstracting out meaningful ideas that hold more generally in large systems, is well-entrenched in the field of information theory, and can become a useful export in fields such as neuroscience.
Acknowledgments
We have many people to thank for extremely useful discussions. A non-exhaustive list follows: Mayank Bakshi, Marlene Behrmann, Todd Coleman, Uday Jagadisan, Haewon Jeong, Rob Kass, Gabe Schamburg, Tsachy Weissman. We also thank the anonymous reviewers whose comments improved our exposition substantially.
Praveen Venkatesh was supported, in part, by a Fellowship in Digital Health from the Center for Machine Learning and Health at Carnegie Mellon University. Pulkit Grover was supported, in part, by an NSF CAREER Award.
Appendix A Proof of Proposition 1
Proof of Proposition 1.
() Suppose there exists some that has -information flow. That is,
[TABLE]
Then,
[TABLE]
where (a) follows from the non-negativity of conditional mutual information and (b) from (77). Taking in Definition 5, we see that set has -information flow.
() Next, suppose that the set has -information flow, as per Definition 5. That is, there exists a set such that
[TABLE]
Also, let be any ordering of the nodes in (where ). Then by the chain rule of mutual information,
[TABLE]
By the non-negativity of conditional mutual information, at least one of the terms in the summation must be strictly positive. Let the index of this term be . Hence, there exists and , such that
[TABLE]
In other words, there exists an edge that has -information flow. ∎
Appendix B Proof of Proposition 9
Proof of Proposition 9.
Consider the set of all that have -information flow. That is, must satisfy
[TABLE]
Define
[TABLE]
Then, we claim that and satisfy equations (63) and (64).
First, note that if , then for every , we must have that
[TABLE]
If not, then by (85), which implies that , which is a contradiction. Hence, we see that no edge in has -information flow. Therefore, by Proposition 1, the set has no -information flow. This directly implies the condition in (64).
Next, we claim that if , then for every , if is a set that satisfies
[TABLE]
then satisfies
[TABLE]
Let , so that . Then,
[TABLE]
by (87). So,
[TABLE]
where (a) and (c) follow from the chain rule, (b) and (d) follow from (64), and (e) follows from (89). Thus, condition (63) also holds. ∎
Appendix C Synergistic Information Flow
C.1 Partial Information Decomposition preliminaries
The literature on Partial Information Decomposition seeks to find a decomposition for the mutual information between a message, , and a set of random variables, into several individually meaningful, non-negative terms [50]. For our purposes, it suffices to consider the bivariate case, i.e., the decomposition of into non-negative components. In the bivariate case, it is well-understood how many components there ought to be, and what these quantities intuitively represent, but as yet, there is no consensus on a single set of definitions [50].
There is, however, consensus on a basic set of properties that we expect these components to satisfy. For our purposes, we will only make use of the basic properties stated here, so that any definition of the aforementioned components which satisfies these properties suffices for our theory.
In the bivariate case, the mutual information between and is decomposed into four components: information about which is
(i) unique to and not present in ,
(ii) unique to and not present in ,
(iii) redundantly present in both and , and
(iv) synergistically present in and .
In the notation of [49], the decomposition is written as:
[TABLE]
where the components are ordered exactly as stated above. Note that refers to “shared”, and hence redundant, information, while refers to “complementary”, and hence synergistic, information. We shall continue to use the terms “redundant” and “synergistic”, however, since they are more meaningful in this context. Also, in what follows, we shall assume that and are symmetric in and . This is usually an additional condition that is imposed when defining these quantities, but here, we take it as given.
Given what we want the four components to represent, we would also expect the following to hold:
[TABLE]
As a natural consequence, this means that the conditional mutual information will satisfy:
[TABLE]
Finally, we want each of these components to always be non-negative:
[TABLE]
It is not obvious that a consistent definition of these four quantities which also satisfies the equations stated above even exists, but in fact, additional properties are required to obtain a unique definition. For instance, see [49] for one such development.
As stated before, our theory only relies on the properties stated in this section. As a result, our theorem on the equivalence of information flow definitions holds irrespective of what definition is used, exactly, for synergistic information. It only matters that the definition used satisfies the basic properties presented here.
C.2 Equivalence of information flow definitions
Proof of Proposition 2.
() Suppose the edge has strictly positive -information flow. Then,
[TABLE]
If I\bigl{(}M;X(E_{t})\bigr{)}>0 with in (98), then condition 1 in Definition 6 holds, so nothing remains to be shown. If not, then I\bigl{(}M;X(E_{t})\bigr{)}=0, so (98) implies that there must exist some such that
[TABLE]
However, since I\bigl{(}M;X(E_{t}^{\prime})\bigr{)}=0, we must have UI\bigl{(}M:X(E_{t})\setminus X(\mathcal{E}_{t}^{\prime})\bigr{)}=0 by (95) and (97). Hence,
[TABLE]
So the implication in the forward direction holds.
() For the converse, suppose that has no -information flow. That is,
[TABLE]
Since and are both non-negative by (97), we must have that
[TABLE]
This proves the converse. ∎
Appendix D Miscellaneous Proofs from Section 5
D.1 Proof of Lemma 10
Proof of Lemma 10.
Consider a subset of hidden nodes that is not -relevant. Then, by Definition 14, carries no -information flow in . This means that
[TABLE]
Specifically, taking , we have
[TABLE]
Therefore, by Definition 15, is -derived. Thus, if is not -relevant, it is -derived. Taking the contrapositive, if is not -derived, then it is -relevant. ∎
D.2 Proof of proposition 11
Proof of Proposition 11.
We are given that
[TABLE]
and must prove that the hidden nodes at time , , are not -derived.
First note that, since , we must have
[TABLE]
where the last line follows from the fact that conditional mutual information is non-negative, and from (107).
Next, observe that Local Markovity conditions (Proposition 6) must hold on the entire graph , which consists of both observed and hidden nodes. If we apply the Local Markovity condition to , we have ——, or in other words
[TABLE]
Note that , where is the subset comprising outgoing edges of that go to . Therefore,
[TABLE]
Expanding this conditional mutual information, we get
[TABLE]
So we have
[TABLE]
where the final inequality follows from (111) and the fact that conditional mutual information is non-negative. Finally, since , we have that I\bigl{(}M;X(\mathcal{Q}(\mathcal{H}_{t}))\,\big{|}\,X(\widetilde{\mathcal{E}}_{t})\bigr{)}>0, just as we in equations (108)–(111). Hence, the Markov chain —— does not hold, so by Definition 15, are not -derived. ∎
Appendix E On the Uniqueness of Our Definition of Information Flow
From the perspective of designing an axiomatic framework, it is desirable to find a minimal set of properties that gives rise to a unique definition of information flow. Although Property 1 helped us motivate a definition for information flow, it did not uniquely specify a definition. Indeed, the all-zero function as well as the all-ones function also satisfy the property, although they are not particularly useful definitions of information flow.
In this section, we provide a set of properties that uniquely leads to our definition of information flow. However, we must acknowledge that we arrived at these properties with the benefit of hindsight, after having proved many other properties of our definition. As such, they are mathematically very similar to our definition, and one might feel uncomfortable with the idea of imposing such a set of properties at the very outset. Our goal here is only to begin a discussion in this direction: a search for a more abstract set of properties that leads to a unique definition of information flow would be a worthy endeavour in future.
Property 4**.**
Let be a computational system, and let be an indicator of the presence of information flow about on an edge. That is, , if information about flows on the edge , and otherwise. We now state three conditions must satisfy, which naturally leads to our definition of information flow (Definition 4):
- 4a)
I\bigl{(}M;X(E_{t})\bigr{)}>0\quad\Rightarrow\quad\mathcal{F}_{M}(E_{t})=1, 2. 4b)
\exists\;\mathcal{E}_{t}^{\prime}\subseteq\mathcal{E}_{t}\!\setminus\!\{E_{t}\}\;\text{ s.t. }\;I\bigl{(}M;X(\mathcal{E}_{t}^{\prime})\,\big{|}\,X(E_{t})\bigr{)}>I\bigl{(}M;X(\mathcal{E}_{t}^{\prime})\bigr{)}\quad\Rightarrow\quad\mathcal{F}_{M}(E_{t})=1, 3. 4c)
I\bigl{(}M;X(E_{t})\,\big{|}\,X(\mathcal{E}_{t}^{\prime})\bigr{)}=0\;\forall\;\mathcal{E}_{t}^{\prime}\subseteq\mathcal{E}_{t}\quad\Rightarrow\quad\mathcal{F}_{M}(E_{t})=0.
Property 4a is a very natural and intuitive requirement for information flow. Property 4b states that an edge should be considered to carry information about , if upon conditioning, its transmission increases the information that some set conveys about . Property 4c is reminiscent of the separability property from Proposition 9, and states that if an edge has no dependence with , no matter what other transmission is conditioned upon, then it can carry no information flow about .
Effectively, Property 4a states that if an edge has unique or redundant information about , then it must carry information flow, while Property 4b states that if an edge has synergistic information about along with some other set of transmissions, then it must carry information flow. Finally, Property 4c states that if all three of these components are absent, then that edge carries no information flow. This also explains how, if any one of these three properties is absent, our definition is no longer unique.
As we acknowledged previously, some of these properties could be seen as too restrictive or contrived, and a more abstract set of properties is certainly desirable. Nevertheless, these properties do uniquely identify our definition of information flow.
Proposition 12** (Uniqueness).**
If is an indicator of information flow that satisfies the conditions in Property 4, then if and only if has -information flow, per Definition 4.
Proof.
() Suppose the edge has no -information flow per Definition 4. This directly implies the condition in Property 4c. Hence, . This proves that if , the edge must have -information flow.
() Suppose the edge has -information flow per Definition 4. Then,
[TABLE]
If , I\bigl{(}M;X(E_{t})\bigr{)}>0, so by Property 4a, . If I\bigl{(}M;X(E_{t})\bigr{)}=0, then (117) guarantees the existence of some such that
[TABLE]
where in (a), we simply added I\bigl{(}M;X(\mathcal{E}_{t}^{\prime})\bigr{)} to both sides; in (b) and (c), we used the chain rule in two different ways; and in (d), we used the fact that I\bigl{(}M;X(E_{t})\bigr{)}=0. So, by Property 4b, we have that . This proves the converse. ∎
Remark
It should be noted that Definition 4 only specifies whether or not a given edge has -information flow. It does not quantify this flow. So Proposition 12 demonstrates the uniqueness of our definition up to an unspecified information volume. If we require that the conditions in Property 4 hold, then any quantitative definition of information flow will go to zero at an edge if and only if the -information flow carried by that edge is zero.
Appendix F Miscellaneous Derivations from Section 6
F.1 Derivation of Expressions in the Second FFT Example from Section 6.2
Here, we derive the expressions used in Figure 13. Recall that , where .
[TABLE]
Next, we show that these intermediate values actually yield the expected values of .
[TABLE]
F.2 Derivation of the Markov Chain Failure in Section 6.3
We wish to show that in the canonical example from Section 6.3, —— is not a valid Markov chain. Recall that and . Let denote differential entropy. Then,
[TABLE]
Here, we have used the fact that if is a zero-mean multivariate Gaussian random variable with variance , then its differential entropy is given by [55, Thm. 8.4.1]
[TABLE]
Next, note that since and , has no extra information about , given . This is obvious when we think of as being , where , and it can be shown that Z^{\prime}\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\widehat{M}_{2}:
[TABLE]
Since all variables involved are zero-mean Gaussians, this naturally implies that \widehat{M}_{2}\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z^{\prime}. Thus, from our previous argument, has no extra information about when given , or in other words, —— is a valid Markov chain. Therefore,
[TABLE]
derived in the same way as (143). From (143) and (150), we can conclude that , and therefore
[TABLE]
Thus, the stated Markov chain, ——, cannot hold.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] Praveen Venkatesh, Sanghamitra Dutta, and Pulkit Grover. How should we define information flow in neural circuits? In 2019 IEEE International Symposium on Information Theory (ISIT) , pages 176–180, July 2019.
- 2[2] Jorge Almeida, Anat R. Fintzi, and Bradford Z. Mahon. Tool manipulation knowledge is retrieved by way of the ventral visual object processing pathway. Cortex , 49(9):2334–2344, 2013.
- 3[3] Andrea Brovelli, Mingzhou Ding, Anders Ledberg, Yonghong Chen, Richard Nakamura, and Steven L. Bressler. Beta oscillations in a large-scale sensorimotor cortical network: Directional influences revealed by Granger causality. Proceedings of the National Academy of Sciences , 101(26):9849–9854, 2004.
- 4[4] M. Bar, K. S. Kassam, A. S. Ghuman, J. Boshyan, A. M. Schmid, A. M. Dale, M. S. Hämäläinen, K. Marinkovic, D. L. Schacter, B. R. Rosen, and E. Halgren. Top-down facilitation of visual recognition. Proceedings of the National Academy of Sciences , 103(2):449–454, 2006.
- 5[5] Adam S. Greenberg, Timothy Verstynen, Yu-Chin Chiu, Steven Yantis, Walter Schneider, and Marlene Behrmann. Visuotopic cortical connectivity underlying attention revealed with white-matter tractography. Journal of Neuroscience , 32(8):2773–2782, 2012.
- 6[6] Constance Hammond, Hagai Bergman, and Peter Brown. Pathological synchronization in Parkinson’s disease: networks, models and treatments. Trends in neurosciences , 30(7):357–364, 2007.
- 7[7] Y Smith, MD Bevan, E Shink, and JP Bolam. Microcircuitry of the direct and indirect pathways of the basal ganglia. Neuroscience , 86(2):353–387, 1998.
- 8[8] Anthony A Grace. Gating of information flow within the limbic system and the pathophysiology of schizophrenia. Brain Research Reviews , 31(2-3):330–341, 2000.