DeepServe: Serverless Large Language Model Serving at Scale

TL;DR

DeepServe is a scalable, serverless platform for efficiently serving large language models in cloud environments, addressing resource management, latency, and scalability challenges with innovative design components.

Contribution

The paper introduces DEEPSERVE, a novel serverless AI platform with a request-job-task model, optimized serving engine, tailored scheduling, and scaling techniques for large language models.

Findings

DEEPSERVE scales to 64 instances within seconds.

It reduces cold start latency for large language models.

Operates effectively on a large NPU cluster for over a year.

Abstract

In this paper, we propose DEEPSERVE, a scalable and serverless AI platform designed to efficiently serve large language models (LLMs) at scale in cloud environments. DEEPSERVE addresses key challenges such as resource allocation, serving efficiency, and cold start latencies through four main design components. First, DEEPSERVE uses a simple serverless abstraction called the request-job-task model, which helps manage diverse AI workloads across posttraining and model-serving tasks. Second, DEEPSERVE integrates an in-house serving engine named FLOWSERVE using a microkernel-inspired design, NPU-centric execution, and SPMD-based parallelism to optimize LLM serving. Third, DEEPSERVE includes novel scheduling policies tailored for a configuration with both PD-disaggregated and PD-colocated instances. Fourth, DEEPSERVE includes optimizations such as pre-warmed pods, DRAM pre-loading, and NPU-fork, which allow DEEPSERVE to scale up to 64 instances in seconds. DEEPSERVE has been in production for over a year, operating on a large Ascend NPU cluster and providing industrystandard APIs for fine-tuning, agent serving, and model serving to our customers.