Uncovering Drift in Textual Data: An Unsupervised Method for Detecting and Mitigating Drift in Machine Learning Models

TL;DR

This paper introduces an unsupervised method for detecting and mitigating data drift in machine learning models using kernel-based statistical tests, which improves model performance without requiring human annotations.

Contribution

The paper presents a novel unsupervised approach employing maximum mean discrepancy to detect drift and identify its root causes, enabling proactive model maintenance.

Findings

Effective drift detection via MMD-based statistical test

Identified high-drift data subsets improve retrained model performance

Reduces reliance on human annotation for drift detection

Abstract

Drift in machine learning refers to the phenomenon where the statistical properties of data or context, in which the model operates, change over time leading to a decrease in its performance. Therefore, maintaining a constant monitoring process for machine learning model performance is crucial in order to proactively prevent any potential performance regression. However, supervised drift detection methods require human annotation and consequently lead to a longer time to detect and mitigate the drift. In our proposed unsupervised drift detection method, we follow a two step process. Our first step involves encoding a sample of production data as the target distribution, and the model training data as the reference distribution. In the second step, we employ a kernel-based statistical test that utilizes the maximum mean discrepancy (MMD) distance metric to compare the reference and target distributions and estimate any potential drift. Our method also identifies the subset of production data that is the root cause of the drift. The models retrained using these identified high drift samples show improved performance on online customer experience quality metrics.