Model Stability with Continuous Data Updates

TL;DR

This paper investigates the stability of NLP machine learning models under continuous data updates, highlighting how design choices and training strategies influence model jitter across various tasks.

Contribution

It introduces a methodology for assessing model stability and provides empirical insights into how architecture, input representation, and training strategies affect stability.

Findings

Non-RNN models are more stable than RNNs in classification tasks.

Pre-trained fastText embeddings enhance model stability.

Ensemble and incremental training strategies significantly improve stability.

Abstract

In this paper, we study the "stability" of machine learning (ML) models within the context of larger, complex NLP systems with continuous training data updates. For this study, we propose a methodology for the assessment of model stability (which we refer to as jitter under various experimental conditions. We find that model design choices, including network architecture and input representation, have a critical impact on stability through experiments on four text classification tasks and two sequence labeling tasks. In classification tasks, non-RNN-based models are observed to be more stable than RNN-based ones, while the encoder-decoder model is less stable in sequence labeling tasks. Moreover, input representations based on pre-trained fastText embeddings contribute to more stability than other choices. We also show that two learning strategies -- ensemble models and incremental training -- have a significant influence on stability. We recommend ML model designers account for trade-offs in accuracy and jitter when making modeling choices.