Detecting model drift using polynomial relations

TL;DR

This paper introduces a novel method for detecting data distribution shifts in machine learning models by analyzing the stability of polynomial relations between features using Bayes Factors, enhancing model reliability in changing environments.

Contribution

The paper proposes a new approach that identifies polynomial relations between features and measures their stability with Bayes Factors to detect data drift effectively.

Findings

Successfully detects various types of data drift in simulations.

Uses polynomial relations and Bayes Factors for robust drift detection.

Validated approach improves model monitoring in dynamic data environments.

Abstract

Machine learning models serve critical functions, such as classifying loan applicants as good or bad risks. Each model is trained under the assumption that the data used in training and in the field come from the same underlying unknown distribution. Often, this assumption is broken in practice. It is desirable to identify when this occurs, to minimize the impact on model performance. We suggest a new approach to detecting change in the data distribution by identifying polynomial relations between the data features. We measure the strength of each identified relation using its R-square value. A strong polynomial relation captures a significant trait of the data which should remain stable if the data distribution does not change. We thus use a set of learned strong polynomial relations to identify drift. For a set of polynomial relations that are stronger than a given threshold, we calculate the amount of drift observed for that relation. The amount of drift is measured by calculating the Bayes Factor for the polynomial relation likelihood of the baseline data versus field data. We empirically validate the approach by simulating a range of changes, and identify drift using the Bayes Factor of the polynomial relation likelihood change.