Sufficient and Necessary Explanations (and What Lies in Between)

TL;DR

This paper explores formal definitions of feature importance in machine learning, highlighting limitations of existing notions and proposing a unified approach that captures a continuum between sufficiency and necessity, improving interpretability.

Contribution

It introduces a unified importance measure that bridges sufficiency and necessity, addressing limitations of existing explanation methods and enhancing feature importance detection.

Findings

Unified importance measure relates to Shapley values and conditional independence.

The approach detects important features missed by traditional methods.

The continuum perspective improves understanding of feature importance.

Abstract

As complex machine learning models continue to find applications in high-stakes decision-making scenarios, it is crucial that we can explain and understand their predictions. Post-hoc explanation methods provide useful insights by identifying important features in an input $\mathbf{x}$ with respect to the model output $f(\mathbf{x})$. In this work, we formalize and study two precise notions of feature importance for general machine learning models: sufficiency and necessity. We demonstrate how these two types of explanations, albeit intuitive and simple, can fall short in providing a complete picture of which features a model finds important. To this end, we propose a unified notion of importance that circumvents these limitations by exploring a continuum along a necessity-sufficiency axis. Our unified notion, we show, has strong ties to other popular definitions of feature importance, like those based on conditional independence and game-theoretic quantities like Shapley values. Crucially, we demonstrate how a unified perspective allows us to detect important features that could be missed by either of the previous approaches alone.