SkyServe: Serving AI Models across Regions and Clouds with Spot Instances

TL;DR

SkyServe is a system that efficiently uses spot and on-demand cloud resources across regions and clouds to reduce costs and improve AI model serving reliability and latency.

Contribution

It introduces SpotHedge, a novel policy for spreading spot replicas across failure domains to enhance availability and cost-efficiency in AI model serving.

Findings

Reduces AI serving costs by 43% on average.

Improves latency at P50, P90, and P99 by over 2 times.

Achieves high resource availability with mixed spot and on-demand replicas.

Abstract

Recent years have witnessed an explosive growth of AI models. The high cost of hosting AI services on GPUs and their demanding service requirements, make it timely and challenging to lower service costs and guarantee service quality. While spot instances have long been offered with a large discount, spot preemptions have discouraged users from using them to host model replicas when serving AI models. To address this, we propose a simple yet efficient policy, SpotHedge, that leverages spot replicas across different failure domains (e.g., regions and clouds) to ensure availability, lower costs, and high service quality. SpotHedge intelligently spreads spot replicas across different regions and clouds to improve availability and reduce correlated preemptions, overprovisions cheap spot replicas than required as a safeguard against possible preemptions, and dynamically falls back to on-demand replicas when spot replicas become unavailable. We built SkyServe, a system leveraging SpotHedge to efficiently serve AI models over a mixture of spot and on-demand replicas across regions and clouds. We compared SkyServe with both research and production systems on real AI workloads: SkyServe reduces cost by 43% on average while achieving high resource availability compared to using on-demand replicas. Additionally, SkyServe improves P50, P90, and P99 latency by 2.3$\times$, 2.1$\times$, 2.1$\times$ on average compared to other research and production systems.