When to retrain a machine learning model

TL;DR

This paper introduces a principled, uncertainty-based approach for determining optimal retraining times of machine learning models amid evolving data, outperforming existing methods across multiple datasets.

Contribution

It proposes a comprehensive formulation of the retraining decision problem and an uncertainty-driven method that forecasts model performance evolution for better timing decisions.

Findings

Outperforms existing baselines on 7 datasets

Effectively detects when to retrain models under distribution shift

Provides a practical solution for real-world model maintenance

Abstract

A significant challenge in maintaining real-world machine learning models is responding to the continuous and unpredictable evolution of data. Most practitioners are faced with the difficult question: when should I retrain or update my machine learning model? This seemingly straightforward problem is particularly challenging for three reasons: 1) decisions must be made based on very limited information - we usually have access to only a few examples, 2) the nature, extent, and impact of the distribution shift are unknown, and 3) it involves specifying a cost ratio between retraining and poor performance, which can be hard to characterize. Existing works address certain aspects of this problem, but none offer a comprehensive solution. Distribution shift detection falls short as it cannot account for the cost trade-off; the scarcity of the data, paired with its unusual structure, makes it a poor fit for existing offline reinforcement learning methods, and the online learning formulation overlooks key practical considerations. To address this, we present a principled formulation of the retraining problem and propose an uncertainty-based method that makes decisions by continually forecasting the evolution of model performance evaluated with a bounded metric. Our experiments addressing classification tasks show that the method consistently outperforms existing baselines on 7 datasets.