Monitoring Machine Learning Models: Online Detection of Relevant Deviations

TL;DR

This paper introduces a sequential monitoring method for machine learning models that effectively detects significant performance degradations while minimizing false alarms, ensuring model reliability in changing environments.

Contribution

It proposes a novel monitoring scheme that accounts for temporal dependence and thresholds, improving detection of relevant changes over existing benchmark methods.

Findings

Outperforms benchmark methods in detecting relevant model quality changes

Reduces unnecessary alerts and false positives

Validated with simulated and real data

Abstract

Machine learning models are essential tools in various domains, but their performance can degrade over time due to changes in data distribution or other factors. On one hand, detecting and addressing such degradations is crucial for maintaining the models' reliability. On the other hand, given enough data, any arbitrary small change of quality can be detected. As interventions, such as model re-training or replacement, can be expensive, we argue that they should only be carried out when changes exceed a given threshold. We propose a sequential monitoring scheme to detect these relevant changes. The proposed method reduces unnecessary alerts and overcomes the multiple testing problem by accounting for temporal dependence of the measured model quality. Conditions for consistency and specified asymptotic levels are provided. Empirical validation using simulated and real data demonstrates the superiority of our approach in detecting relevant changes in model quality compared to benchmark methods. Our research contributes a practical solution for distinguishing between minor fluctuations and meaningful degradations in machine learning model performance, ensuring their reliability in dynamic environments.