Score-Based Change Detection for Gradient-Based Learning Machines

TL;DR

This paper introduces a versatile score-based change detection method for monitoring machine learning models trained via empirical risk minimization, capable of detecting changes in multiple model components in real-time.

Contribution

It proposes a generic, statistically consistent change detection algorithm applicable within differentiable programming frameworks, with practical calibration for false alarm control.

Findings

Effective on synthetic data

Validated on real-world datasets

Achieves controlled false alarm rate

Abstract

The widespread use of machine learning algorithms calls for automatic change detection algorithms to monitor their behavior over time. As a machine learning algorithm learns from a continuous, possibly evolving, stream of data, it is desirable and often critical to supplement it with a companion change detection algorithm to facilitate its monitoring and control. We present a generic score-based change detection method that can detect a change in any number of components of a machine learning model trained via empirical risk minimization. This proposed statistical hypothesis test can be readily implemented for such models designed within a differentiable programming framework. We establish the consistency of the hypothesis test and show how to calibrate it to achieve a prescribed false alarm rate. We illustrate the versatility of the approach on synthetic and real data.