On shared and multiple information

TL;DR

This paper tackles three key issues in information theory: decomposing information from multiple sources, measuring shared information among sources, and extending mutual information to multiple variables.

Contribution

It proposes new definitions for information decomposition, shared information, and multiple mutual information, advancing the theoretical framework of information theory.

Findings

Provides formal definitions for information decomposition

Introduces measures for shared information among sources

Extends mutual information to multiple variables

Abstract

We address three outstanding problems in information theory. Problem one is the definition of a non-negative decomposition of the information conveyed by two or more sources about a target variable into the specific contribution of each possible combination of the sources. Problem two is the definition of a measure of information shared by several sources about the target variable. Problem three is the definition of a measure of multiple mutual information, that is, the extension of mutual information to more than two variables.