CascadeServe: Unlocking Model Cascades for Inference Serving

TL;DR

CascadeServe introduces an automated system that leverages model cascades for efficient, cost-effective inference serving, adapting dynamically to workload variations while maintaining accuracy.

Contribution

It presents CascadeServe, the first system to integrate model cascades into online inference serving with automated offline planning and real-time adaptation.

Findings

Achieves 2-3x cost savings over baselines

Effectively adapts to workload variations with minimal overhead

Maintains high accuracy while reducing computational costs

Abstract

Machine learning (ML) models are increasingly deployed to production, calling for efficient inference serving systems. Efficient inference serving is complicated by two challenges: (i) ML models incur high computational costs, and (ii) the request arrival rates of practical applications have frequent, high, and sudden variations which make it hard to correctly provision hardware. Model cascades are positioned to tackle both of these challenges, as they (i) save work while maintaining accuracy, and (ii) expose a high-resolution trade-off between work and accuracy, allowing for fine-grained adjustments to request arrival rates. Despite their potential, model cascades haven't been used inside an online serving system. This comes with its own set of challenges, including workload adaption, model replication onto hardware, inference scheduling, request batching, and more. In this work, we propose CascadeServe, which automates and optimizes end-to-end inference serving with cascades. CascadeServe operates in an offline and online phase. In the offline phase, the system pre-computes a gear plan that specifies how to serve inferences online. In the online phase, the gear plan allows the system to serve inferences while making near-optimal adaptations to the query load at negligible decision overheads. We find that CascadeServe saves 2-3x in cost across a wide spectrum of the latency-accuracy space when compared to state-of-the-art baselines on different workloads.