"Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts

TL;DR

This paper introduces a method to attribute changes in machine learning model performance to specific distribution shifts in data, using a game-theoretic approach with Shapley values, validated on various datasets.

Contribution

It formulates performance attribution as a cooperative game and derives an importance weighting method to quantify each distribution's impact, a novel approach for understanding model failures.

Findings

Effective in attributing performance drops to distribution shifts

Validated on synthetic, semi-synthetic, and real-world data

Provides insights for diagnosing model failures

Abstract

Machine learning models frequently experience performance drops under distribution shifts. The underlying cause of such shifts may be multiple simultaneous factors such as changes in data quality, differences in specific covariate distributions, or changes in the relationship between label and features. When a model does fail during deployment, attributing performance change to these factors is critical for the model developer to identify the root cause and take mitigating actions. In this work, we introduce the problem of attributing performance differences between environments to distribution shifts in the underlying data generating mechanisms. We formulate the problem as a cooperative game where the players are distributions. We define the value of a set of distributions to be the change in model performance when only this set of distributions has changed between environments, and derive an importance weighting method for computing the value of an arbitrary set of distributions. The contribution of each distribution to the total performance change is then quantified as its Shapley value. We demonstrate the correctness and utility of our method on synthetic, semi-synthetic, and real-world case studies, showing its effectiveness in attributing performance changes to a wide range of distribution shifts.