A Tale of Two Scales: Reconciling Horizontal and Vertical Scaling for Inference Serving Systems

TL;DR

This paper presents Themis, a hybrid autoscaling system for inference serving that combines horizontal and vertical scaling strategies to improve performance and resource efficiency under varying workloads.

Contribution

Themis introduces a two-stage autoscaling approach that dynamically switches between vertical and horizontal scaling based on workload conditions, optimizing inference system performance.

Findings

Achieves over 10x reduction in SLO violations compared to existing methods.

Effectively balances resource efficiency and performance under real-world workloads.

Demonstrates significant improvements in inference serving systems through extensive evaluations.

Abstract

Inference serving is of great importance in deploying machine learning models in real-world applications, ensuring efficient processing and quick responses to inference requests. However, managing resources in these systems poses significant challenges, particularly in maintaining performance under varying and unpredictable workloads. Two primary scaling strategies, horizontal and vertical scaling, offer different advantages and limitations. Horizontal scaling adds more instances to handle increased loads but can suffer from cold start issues and increased management complexity. Vertical scaling boosts the capacity of existing instances, allowing for quicker responses but is limited by hardware and model parallelization capabilities. This paper introduces Themis, a system designed to leverage the benefits of both horizontal and vertical scaling in inference serving systems. Themis employs a two-stage autoscaling strategy: initially using in-place vertical scaling to handle workload surges and then switching to horizontal scaling to optimize resource efficiency once the workload stabilizes. The system profiles the processing latency of deep learning models, calculates queuing delays, and employs different dynamic programming algorithms to solve the joint horizontal and vertical scaling problem optimally based on the workload situation. Extensive evaluations with real-world workload traces demonstrate over $10\times$ SLO violation reduction compared to the state-of-the-art horizontal or vertical autoscaling approaches while maintaining resource efficiency when the workload is stable.