Trustworthy Representation Learning via Information Funnels and Bottlenecks

TL;DR

This paper introduces a novel information-theoretic framework, CPFSI, for learning invariant, fair, and private representations in machine learning, demonstrating its effectiveness and real-world applicability especially in data-scarce tabular settings.

Contribution

The paper proposes the Conditional Privacy Funnel with Side-information (CPFSI), a new approach within information bottleneck methods, with neural approximations and analysis of trade-offs for fair, private, and invariant representations.

Findings

CPFSI effectively balances utility, fairness, and privacy.

Intervening on sensitive attributes improves fairness without sacrificing performance.

Method outperforms existing approaches in real-world tabular datasets.

Abstract

Ensuring trustworthiness in machine learning -- by balancing utility, fairness, and privacy -- remains a critical challenge, particularly in representation learning. In this work, we investigate a family of closely related information-theoretic objectives, including information funnels and bottlenecks, designed to extract invariant representations from data. We introduce the Conditional Privacy Funnel with Side-information (CPFSI), a novel formulation within this family, applicable in both fully and semi-supervised settings. Given the intractability of these objectives, we derive neural-network-based approximations via amortized variational inference. We systematically analyze the trade-offs between utility, invariance, and representation fidelity, offering new insights into the Pareto frontiers of these methods. Our results demonstrate that CPFSI effectively balances these competing objectives and frequently outperforms existing approaches. Furthermore, we show that by intervening on sensitive attributes in CPFSI's predictive posterior enhances fairness while maintaining predictive performance. Finally, we focus on the real-world applicability of these approaches, particularly for learning robust and fair representations from tabular datasets in data scarce-environments -- a modality where these methods are often especially relevant.