Conformal Prediction Under Distribution Shift: A COVID-19 Natural Experiment

TL;DR

This study investigates how conformal prediction methods perform under distribution shifts, specifically during COVID-19, revealing that feature importance concentration impacts robustness and that regular retraining can mitigate coverage loss.

Contribution

The paper introduces a framework for assessing conformal prediction robustness under distribution shifts using SHAP analysis and provides practical guidelines for monitoring and retraining.

Findings

Coverage drops vary widely despite feature turnover.

Feature importance concentration correlates with catastrophic failures.

Quarterly retraining improves coverage for vulnerable tasks.

Abstract

Conformal prediction guarantees degrade under distribution shift. We study this using COVID-19 as a natural experiment across 8 supply chain tasks. Despite identical severe feature turnover (Jaccard approximately 0), coverage drops vary from 0% to 86.7%, spanning two orders of magnitude. Using SHapley Additive exPlanations (SHAP) analysis, we find catastrophic failures correlate with single-feature dependence (rho = 0.714, p = 0.047). Catastrophic tasks concentrate importance in one feature (4.5x increase), while robust tasks redistribute across many (10-20x). Quarterly retraining restores catastrophic task coverage from 22% to 41% (+19 pp, p = 0.04), but provides no benefit for robust tasks (99.8% coverage). Exploratory analysis of 4 additional tasks with moderate feature stability (Jaccard 0.13-0.86) reveals feature stability, not concentration, determines robustness, suggesting concentration effects apply specifically to severe shifts. We provide a decision framework: monitor SHAP concentration before deployment; retrain quarterly if vulnerable (>40% concentration); skip retraining if robust.