Ensembling Shift Detectors: an Extensive Empirical Evaluation

TL;DR

This paper introduces an ensemble approach for shift detection that combines multiple detectors and tunes their significance levels, resulting in a more robust method capable of identifying various dataset shifts in real-world scenarios.

Contribution

It proposes a novel ensemble technique for shift detectors that adapts to different shift types, improving robustness over existing single-detector methods.

Findings

Ensemble shift detectors outperform individual detectors in diverse shift scenarios.

The method is validated through extensive benchmark experiments on real-world datasets.

Tuning significance levels enhances detection accuracy across shift types.

Abstract

The term dataset shift refers to the situation where the data used to train a machine learning model is different from where the model operates. While several types of shifts naturally occur, existing shift detectors are usually designed to address only a specific type of shift. We propose a simple yet powerful technique to ensemble complementary shift detectors, while tuning the significance level of each detector's statistical test to the dataset. This enables a more robust shift detection, capable of addressing all different types of shift, which is essential in real-life settings where the precise shift type is often unknown. This approach is validated by a large-scale statistically sound benchmark study over various synthetic shifts applied to real-world structured datasets.