Information Subtraction: Learning Representations for Conditional Entropy

TL;DR

This paper introduces Information Subtraction, a flexible framework for learning representations that selectively preserve desired information while removing undesired aspects, applicable to continuous variables and useful for fair learning and domain generalization.

Contribution

It extends conditional entropy representation learning to continuous variables using a generative architecture that disentangles and subtracts specific information components.

Findings

Effective in fair learning scenarios

Improves domain generalization

Provides semantic features of conditional entropy

Abstract

The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed information regarding discrete sensitive variables, they have not yet extended their scope to continuous cases. This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. With its flexibility in disentangling information, we can iteratively apply Information Subtraction to represent arbitrary information components between continuous variables, thereby explaining the various relationships that exist between them. Our results highlight the representations' ability to provide semantic features of conditional entropy. By subtracting sensitive and domain-specific information, our framework demonstrates effective performance in fair learning and domain generalization. The code for this paper is available at https://github.com/jh-liang/Information-Subtraction