Training and Serving Machine Learning Models at Scale

TL;DR

This paper discusses the challenges of managing machine learning models during training and inference in web services, proposing initial solutions that improve efficiency and predictability with minimal user input.

Contribution

It introduces initial solutions for managing ML services during training and inference, addressing complexity and quality requirements.

Findings

Solutions improve system efficiency

Enhance predictability of response time and accuracy

Reduce manual management efforts

Abstract

In recent years, Web services are becoming more and more intelligent (e.g., in understanding user preferences) thanks to the integration of components that rely on Machine Learning (ML). Before users can interact (inference phase) with an ML-based service (ML-Service), the underlying ML model must learn (training phase) from existing data, a process that requires long-lasting batch computations. The management of these two, diverse phases is complex and meeting time and quality requirements can hardly be done with manual approaches. This paper highlights some of the major issues in managing ML-services in both training and inference modes and presents some initial solutions that are able to meet set requirements with minimum user inputs. A preliminary evaluation demonstrates that our solutions allow these systems to become more efficient and predictable with respect to their response time and accuracy.