A unified framework for dataset shift diagnostics

TL;DR

DetectShift is a flexible framework that quantifies and tests for various dataset shifts across different data types, aiding practitioners in model adaptation especially when target labels are scarce.

Contribution

Introduces DetectShift, a novel framework for diagnosing multiple types of dataset shifts in diverse data modalities, enhancing model robustness.

Findings

Effectively detects dataset shifts in high-dimensional data

Quantifies magnitude of shifts for better interpretability

Applicable to regression, classification, and various data forms

Abstract

Supervised learning techniques typically assume training data originates from the target population. Yet, in reality, dataset shift frequently arises, which, if not adequately taken into account, may decrease the performance of their predictors. In this work, we propose a novel and flexible framework called DetectShift that quantifies and tests for multiple dataset shifts, encompassing shifts in the distributions of $(X, Y)$, $X$, $Y$, $X|Y$, and $Y|X$. DetectShift equips practitioners with insights into data shifts, facilitating the adaptation or retraining of predictors using both source and target data. This proves extremely valuable when labeled samples in the target domain are limited. The framework utilizes test statistics with the same nature to quantify the magnitude of the various shifts, making results more interpretable. It is versatile, suitable for regression and classification tasks, and accommodates diverse data forms - tabular, text, or image. Experimental results demonstrate the effectiveness of DetectShift in detecting dataset shifts even in higher dimensions.