Desiderata for next generation of ML model serving

TL;DR

This paper discusses essential qualities for future ML inference platforms, emphasizing data-centric design to improve deployment and scalability of ML systems.

Contribution

It identifies key qualities for next-gen inference systems and advocates for data-centricity as a core design principle.

Findings

Highlights importance of data-centricity in ML inference

Proposes practical approaches for achieving desired qualities

Provides rationale for each quality in inference platform design

Abstract

Inference is a significant part of ML software infrastructure. Despite the variety of inference frameworks available, the field as a whole can be considered in its early days. This position paper puts forth a range of important qualities that next generation of inference platforms should be aiming for. We present our rationale for the importance of each quality, and discuss ways to achieve it in practice. We propose to focus on data-centricity as the overarching design pattern which enables smarter ML system deployment and operation at scale.