HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing

TL;DR

HybridServe is a system that efficiently serves large AI models by dynamically choosing smaller or larger models based on confidence, significantly reducing energy consumption while maintaining accuracy.

Contribution

It introduces a confidence-based hybrid serving approach and a dataflow planner to optimize energy efficiency and throughput in large DNN serving systems.

Findings

Reduces energy footprint by up to 19.8x compared to state-of-the-art systems.

Maintains accuracy comparable to using only giant DNNs.

Improves system throughput through optimized model partitioning.

Abstract

Giant Deep Neural Networks (DNNs), have become indispensable for accurate and robust support of large-scale cloud based AI services. However, serving giant DNNs is prohibitively expensive from an energy consumption viewpoint easily exceeding that of training, due to the enormous scale of GPU clusters needed to hold giant DNN model partitions and replicas. Existing approaches can either optimize energy efficiency or inference accuracy but not both. To overcome this status quo, we propose HybridServe, a novel hybrid DNN model serving system that leverages multiple sized versions (small to giant) of the model to be served in tandem. Through a confidence based hybrid model serving dataflow, HybridServe prefers to serve inference requests with energy-efficient smaller models so long as accuracy is not compromised, thereby reducing the number of replicas needed for giant DNNs. HybridServe also features a dataflow planner for efficient partitioning and replication of candidate models to maximize serving system throughput. Experimental results using a prototype implementation of HybridServe show that it reduces energy footprint by up to 19.8x compared to the state-of-the-art DNN model serving systems while matching the accuracy of serving solely with giant DNNs.