ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systems

TL;DR

ModelCI-e is a lightweight plugin that facilitates continual learning and seamless model updates in deep learning serving systems, addressing the challenge of dynamic data environments in MLOps.

Contribution

It introduces a comprehensive MLOps plugin with a model factory, CL backend, and web interface, enabling efficient model updating without modifying serving engines.

Findings

ModelCI-e simplifies prototyping and benchmarking of CL models.

Eliminating interference between updating and inference improves system efficiency.

Preliminary results show the usability of ModelCI-e in real-world scenarios.

Abstract

MLOps is about taking experimental ML models to production, i.e., serving the models to actual users. Unfortunately, existing ML serving systems do not adequately handle the dynamic environments in which online data diverges from offline training data, resulting in tedious model updating and deployment works. This paper implements a lightweight MLOps plugin, termed ModelCI-e (continuous integration and evolution), to address the issue. Specifically, it embraces continual learning (CL) and ML deployment techniques, providing end-to-end supports for model updating and validation without serving engine customization. ModelCI-e includes 1) a model factory that allows CL researchers to prototype and benchmark CL models with ease, 2) a CL backend to automate and orchestrate the model updating efficiently, and 3) a web interface for an ML team to manage CL service collaboratively. Our preliminary results demonstrate the usability of ModelCI-e, and indicate that eliminating the interference between model updating and inference workloads is crucial for higher system efficiency.