Self-Updating Models with Error Remediation

TL;DR

The paper introduces SUMER, a framework enabling deployed machine learning models to self-update using semi-supervised learning and error remediation, improving performance on unseen data without extensive retraining.

Contribution

The paper presents SUMER, a novel framework that allows models to self-update with error remediation, addressing challenges of updating deployed models in dynamic environments.

Findings

SUMER outperforms non-updating models on unseen data.

Self-updating models perform better with limited initial training data.

Error remediation enhances the effectiveness of self-updating models.

Abstract

Many environments currently employ machine learning models for data processing and analytics that were built using a limited number of training data points. Once deployed, the models are exposed to significant amounts of previously-unseen data, not all of which is representative of the original, limited training data. However, updating these deployed models can be difficult due to logistical, bandwidth, time, hardware, and/or data sensitivity constraints. We propose a framework, Self-Updating Models with Error Remediation (SUMER), in which a deployed model updates itself as new data becomes available. SUMER uses techniques from semi-supervised learning and noise remediation to iteratively retrain a deployed model using intelligently-chosen predictions from the model as the labels for new training iterations. A key component of SUMER is the notion of error remediation as self-labeled data can be susceptible to the propagation of errors. We investigate the use of SUMER across various data sets and iterations. We find that self-updating models (SUMs) generally perform better than models that do not attempt to self-update when presented with additional previously-unseen data. This performance gap is accentuated in cases where there is only limited amounts of initial training data. We also find that the performance of SUMER is generally better than the performance of SUMs, demonstrating a benefit in applying error remediation. Consequently, SUMER can autonomously enhance the operational capabilities of existing data processing systems by intelligently updating models in dynamic environments.