Concepts' Information Bottleneck Models

TL;DR

This paper introduces an information bottleneck regularizer for concept bottleneck models, improving their accuracy, interpretability, and faithfulness by encouraging minimal, sufficient concept representations without changing architecture.

Contribution

It proposes a novel IB-based regularizer for CBMs, applicable across architectures, that enhances performance and interpretability by controlling information flow in the concept layer.

Findings

Consistent performance improvements across six CBM families.

Enhanced interpretability and faithfulness of concept representations.

Robust gains demonstrated on multiple benchmarks.

Abstract

Concept Bottleneck Models (CBMs) aim to deliver interpretable predictions by routing decisions through a human-understandable concept layer, yet they often suffer reduced accuracy and concept leakage that undermines faithfulness. We introduce an explicit Information Bottleneck regularizer on the concept layer that penalizes $I(X;C)$ while preserving task-relevant information in $I(C;Y)$, encouraging minimal-sufficient concept representations. We derive two practical variants (a variational objective and an entropy-based surrogate) and integrate them into standard CBM training without architectural changes or additional supervision. Evaluated across six CBM families and three benchmarks, the IB-regularized models consistently outperform their vanilla counterparts. Information-plane analyses further corroborate the intended behavior. These results indicate that enforcing a minimal-sufficient concept bottleneck improves both predictive performance and the reliability of concept-level interventions. The proposed regularizer offers a theoretic-grounded, architecture-agnostic path to more faithful and intervenable CBMs, resolving prior evaluation inconsistencies by aligning training protocols and demonstrating robust gains across model families and datasets.