Optimizing Prediction Serving on Low-Latency Serverless Dataflow

TL;DR

This paper introduces Cloudflow, a serverless dataflow system optimized for low-latency prediction serving, achieving up to 2x performance improvements over existing systems through operator fusion and execution optimizations.

Contribution

The paper presents Cloudflow, a novel serverless dataflow system that optimizes prediction serving with a familiar API and performance enhancements without modifying ML models.

Findings

Cloudflow achieves up to 2x performance gains over state-of-the-art systems.

Operator fusion and optimized execution are key to performance improvements.

Cloudflow successfully meets latency goals in real-time prediction tasks.

Abstract

Prediction serving systems are designed to provide large volumes of low-latency inferences machine learning models. These systems mix data processing and computationally intensive model inference and benefit from multiple heterogeneous processors and distributed computing resources. In this paper, we argue that a familiar dataflow API is well-suited to this latency-sensitive task, and amenable to optimization even with unmodified black-box ML models. We present the design of Cloudflow, a system that provides this API and realizes it on an autoscaling serverless backend. Cloudflow transparently implements performance-critical optimizations including operator fusion and competitive execution. Our evaluation shows that Cloudflow's optimizations yield significant performance improvements on synthetic workloads and that Cloudflow outperforms state-of-the-art prediction serving systems by as much as 2x on real-world prediction pipelines, meeting latency goals of demanding applications like real-time video analysis.