Towards Stable Machine Learning Model Retraining via Slowly Varying Sequences

TL;DR

This paper introduces a model-agnostic framework for stable ML model retraining that balances predictive power and stability, improving interpretability and trust across diverse models and domains.

Contribution

It develops a mixed-integer optimization approach and an efficient algorithm to find Pareto optimal models with enhanced stability and generalization, considering custom stability metrics.

Findings

Achieved a 2% reduction in predictive power with a 30% increase in stability.

Validated framework across multiple models and domains, including healthcare and vision.

Deployed in a hospital setting, demonstrating practical effectiveness.

Abstract

We consider the problem of retraining machine learning (ML) models when new batches of data become available. Existing approaches greedily optimize for predictive power independently at each batch, without considering the stability of the model's structure or analytical insights across retraining iterations. We propose a model-agnostic framework for finding sequences of models that are stable across retraining iterations. We develop a mixed-integer optimization formulation that is guaranteed to recover Pareto optimal models (in terms of the predictive power-stability trade-off) with good generalization properties, as well as an efficient polynomial-time algorithm that performs well in practice. We focus on retaining consistent analytical insights-which is important to model interpretability, ease of implementation, and fostering trust with users-by using custom-defined distance metrics that can be directly incorporated into the optimization problem. We evaluate our framework across models (regression, decision trees, boosted trees, and neural networks) and application domains (healthcare, vision, and language), including deployment in a production pipeline at a major US hospital. We find that, on average, a 2% reduction in predictive power leads to a 30% improvement in stability.