Separable Computation of Information Measures

TL;DR

This paper introduces a separable approach to computing various information measures using learned feature representations, enabling efficient estimation and revealing new links between information measures and dependence structures.

Contribution

It demonstrates that multiple information measures can be computed from learned features under mild conditions, providing theoretical guarantees for practical estimation methods.

Findings

Separable computation applies to mutual information, f-information, and others.

Establishes new connections between information measures and dependence structures.

Provides theoretical guarantees for representation-based estimation methods.

Abstract

We study a separable design for computing information measures, where the information measure is computed from learned feature representations instead of raw data. Under mild assumptions on the feature representations, we demonstrate that a class of information measures admit such separable computation, including mutual information, $f$-information, Wyner's common information, G{\'a}cs--K{\"o}rner common information, and Tishby's information bottleneck. Our development establishes several new connections between information measures and the statistical dependence structure. The characterizations also provide theoretical guarantees of practical designs for estimating information measures through representation learning.