TRACE: Theoretical Risk Attribution under Covariate-shift Effects

TL;DR

TRACE provides a theoretical framework to diagnose and quantify the impact of covariate shift on model performance, enabling safer model replacement decisions through interpretable risk attribution.

Contribution

We introduce TRACE, a novel framework that decomposes risk change under covariate shift into interpretable components and provides practical diagnostics for model deployment.

Findings

TRACE accurately captures risk change in linear regression.

Diagnostics correlate strongly with true performance degradation.

Deployment gate score effectively guides model replacement decisions.

Abstract

When a source-trained model $Q$ is replaced by a model $\tilde{Q}$ trained on shifted data, its performance on the source domain can change unpredictably. To address this, we study the two-model risk change, $\Delta R := R_P(\tilde{Q}) - R_P(Q)$, under covariate shift. We introduce TRACE (Theoretical Risk Attribution under Covariate-shift Effects), a framework that decomposes $|\Delta R|$ into an interpretable upper bound. This decomposition disentangles the risk change into four actionable factors: two generalization gaps, a model change penalty, and a covariate shift penalty, transforming the bound into a powerful diagnostic tool for understanding why performance has changed. To make TRACE a fully computable diagnostic, we instantiate each term. The covariate shift penalty is estimated via a model sensitivity factor (from high-quantile input gradients) and a data-shift measure; we use feature-space Optimal Transport (OT) by default and provide a robust alternative using Maximum Mean Discrepancy (MMD). The model change penalty is controlled by the average output distance between the two models on the target sample. Generalization gaps are estimated on held-out data. We validate our framework in an idealized linear regression setting, showing the TRACE bound correctly captures the scaling of the true risk difference with the magnitude of the shift. Across synthetic and vision benchmarks, TRACE diagnostics are valid and maintain a strong monotonic relationship with the true performance degradation. Crucially, we derive a deployment gate score that correlates strongly with $|\Delta R|$ and achieves high AUROC/AUPRC for gating decisions, enabling safe, label-efficient model replacement.