Extracting Unique Information Through Markov Relations

TL;DR

This paper introduces two new Markov-based measures for extracting the unique information that variable X provides about a message M, distinct from Y, with applications in fair machine learning and information theory.

Contribution

It proposes novel measures for unique information extraction based on Markov relations and characterizes them in the Gaussian case, connecting to the PID framework.

Findings

Complete characterization in the Gaussian case

Measures achieve non-negativity within PID

Non-symmetry property observed in the measures

Abstract

We propose two new measures for extracting the unique information in $X$ and not $Y$ about a message $M$, when $X, Y$ and $M$ are joint random variables with a given joint distribution. We take a Markov based approach, motivated by questions in fair machine learning, and inspired by similar Markov-based optimization problems that have been used in the Information Bottleneck and Common Information frameworks. We obtain a complete characterization of our definitions in the Gaussian case (namely, when $X, Y$ and $M$ are jointly Gaussian), under the assumption of Gaussian optimality. We also examine the consistency of our definitions with the partial information decomposition (PID) framework, and show that these Markov based definitions achieve non-negativity, but not symmetry, within the PID framework.