On the Relationship Between Explanation and Prediction: A Causal View

TL;DR

This paper uses causal inference to systematically analyze how various upstream factors influence explanations of machine learning models, revealing that explanations often poorly reflect the actual predictions, especially in high-performing models.

Contribution

It introduces a causal framework to quantify the relationship between explanations and predictions, highlighting limitations of current explanation methods.

Findings

Explanations often have a weak relationship with model predictions.

Higher-performing models show larger gaps between explanations and predictions.

The study provides a foundation for developing quantitative metrics for explanations.

Abstract

Being able to provide explanations for a model's decision has become a central requirement for the development, deployment, and adoption of machine learning models. However, we are yet to understand what explanation methods can and cannot do. How do upstream factors such as data, model prediction, hyperparameters, and random initialization influence downstream explanations? While previous work raised concerns that explanations (E) may have little relationship with the prediction (Y), there is a lack of conclusive study to quantify this relationship. Our work borrows tools from causal inference to systematically assay this relationship. More specifically, we study the relationship between E and Y by measuring the treatment effect when intervening on their causal ancestors, i.e., on hyperparameters and inputs used to generate saliency-based Es or Ys. Our results suggest that the relationships between E and Y is far from ideal. In fact, the gap between 'ideal' case only increase in higher-performing models -- models that are likely to be deployed. Our work is a promising first step towards providing a quantitative measure of the relationship between E and Y, which could also inform the future development of methods for E with a quantitative metric.