A unified theory of information transfer and causal relation

TL;DR

This paper develops a comprehensive theoretical framework linking information transfer and causal relations, revealing their origins in high-order mutual information phenomena and bridging concepts from physics and computer science.

Contribution

It introduces a unified theory explaining how information transfer and causality originate, differ, and relate through high-order mutual information, addressing foundational questions.

Findings

Information transfer and causal relation originate from high-order mutual information phenomena.

The theory explains mechanisms for the emergence, disappearance, and differences of information transfer and causality.

Effect sizes of information transfer and causal relation are defined based on high-dimensional coupling events.

Abstract

Information transfer between coupled stochastic dynamics, measured by transfer entropy and information flow, is suggested as a physical process underlying the causal relation of systems. While information transfer analysis has booming applications in both science and engineering fields, critical mysteries about its foundations remain unsolved. Fundamental yet difficult questions concern how information transfer and causal relation originate, what they depend on, how they differ from each other, and if they are created by a unified and general quantity. These questions essentially determine the validity of causal relation measurement via information transfer. Here we pursue to lay a complete theoretical basis of information transfer and causal relation. Beyond the well-known relations between these concepts that conditionally hold, we demonstrate that information transfer and causal relation universally originate from specific information synergy and redundancy phenomena characterized by high-order mutual information. More importantly, our theory analytically explains the mechanisms for information transfer and causal relation to originate, vanish, and differ from each other. Moreover, our theory naturally defines the effect sizes of information transfer and causal relation based on high-dimensional coupling events. These results may provide a unified view of information, synergy, and causal relation to bridge Pearl's causal inference theory in computer science and information transfer analysis in physics.