Mitigating Bias in Concept Bottleneck Models for Fair and Interpretable Image Classification

TL;DR

This paper introduces three techniques to reduce bias in concept bottleneck models for image classification, significantly improving fairness while maintaining interpretability.

Contribution

It proposes novel bias mitigation methods for CBMs, including concept filtering, removal, and adversarial training, enhancing fairness without sacrificing accuracy.

Findings

Outperforms prior methods in fairness-performance tradeoffs

Reduces gender bias on datasets like ImSitu

Improves interpretability and fairness in image classification

Abstract

Ensuring fairness in image classification prevents models from perpetuating and amplifying bias. Concept bottleneck models (CBMs) map images to high-level, human-interpretable concepts before making predictions via a sparse, one-layer classifier. This structure enhances interpretability and, in theory, supports fairness by masking sensitive attribute proxies such as facial features. However, CBM concepts have been known to leak information unrelated to concept semantics and early results reveal only marginal reductions in gender bias on datasets like ImSitu. We propose three bias mitigation techniques to improve fairness in CBMs: 1. Decreasing information leakage using a top-k concept filter, 2. Removing biased concepts, and 3. Adversarial debiasing. Our results outperform prior work in terms of fairness-performance tradeoffs, indicating that our debiased CBM provides a significant step towards fair and interpretable image classification.