HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds

TL;DR

HydraServe is a system designed to significantly reduce cold start latency for serverless large language model serving in public clouds by proactive distribution, overlapping stages, strategic placement, and pipeline consolidation.

Contribution

HydraServe introduces novel techniques for proactive model distribution, overlapping cold start stages, strategic GPU placement, and pipeline merging to minimize cold start latency in serverless LLM serving.

Findings

Reduces cold start latency by 1.7× to 4.7×

Improves service level objective attainment by 1.43× to 1.74×

Demonstrates effectiveness across diverse settings

Abstract

With the proliferation of large language model (LLM) variants, developers are turning to serverless computing for cost-efficient LLM deployment. However, public cloud providers often struggle to provide performance guarantees for serverless LLM serving due to significant cold start latency caused by substantial model sizes and complex runtime dependencies. To address this problem, we present HydraServe, a serverless LLM serving system designed to minimize cold start latency in public clouds. HydraServe proactively distributes models across servers to quickly fetch them, and overlaps cold-start stages within workers to reduce startup latency. Additionally, HydraServe strategically places workers across GPUs to avoid network contention among cold-start instances. To minimize resource consumption during cold starts, HydraServe further introduces pipeline consolidation that can merge groups of workers into individual serving endpoints. Our comprehensive evaluations under diverse settings demonstrate that HydraServe reduces the cold start latency by 1.7$\times$-- 4.7$\times$ and improves service level objective attainment by 1.43$\times$--1.74$\times$ compared to baselines.