Promises and Pitfalls of Black-Box Concept Learning Models

TL;DR

This paper investigates the limitations of black-box concept learning models, revealing that they often encode unintended information, which can mislead interpretations despite mitigation efforts.

Contribution

It uncovers the mechanisms of information leakage in concept learning models and proposes strategies to mitigate these issues.

Findings

Concept representations encode unintended information

Mitigation strategies often fail to prevent information leakage

Interpretability of models can be misleading due to hidden information

Abstract

Machine learning models that incorporate concept learning as an intermediate step in their decision making process can match the performance of black-box predictive models while retaining the ability to explain outcomes in human understandable terms. However, we demonstrate that the concept representations learned by these models encode information beyond the pre-defined concepts, and that natural mitigation strategies do not fully work, rendering the interpretation of the downstream prediction misleading. We describe the mechanism underlying the information leakage and suggest recourse for mitigating its effects.