Symphony: Optimized DNN Model Serving using Deferred Batch Scheduling

TL;DR

Symphony is a DNN model serving system that uses deferred batch scheduling to improve efficiency and throughput, achieving higher goodput and better GPU utilization while supporting autoscaling.

Contribution

It introduces a novel deferred batch scheduling approach that optimizes GPU usage and system throughput in DNN inference serving.

Findings

Symphony achieves 5x higher goodput compared to prior systems.

It reduces GPU usage by 60% for the same workload.

The system can schedule millions of requests per second across thousands of GPUs.

Abstract

Having large batch sizes is one of the most critical aspects of increasing the accelerator efficiency and the performance of DNN model inference. However, existing model serving systems cannot achieve adequate batch sizes while meeting latency objectives as these systems eagerly dispatch requests to accelerators to minimize the accelerator idle time. We propose Symphony, a DNN serving system that explores deferred batch scheduling to optimize system efficiency and throughput. Further, unlike other prior systems, Symphony's GPU usage is load-proportional: it consolidates workloads on the appropriate number of GPUs and works smoothly with cluster auto-scaling tools. Symphony consists of two core design points. First, Symphony defines a schedulable window in which a batch of inference requests can be dispatched. This window is computed in order to improve accelerator efficiency while meeting the request's SLO. Second, Symphony implements a scalable, low-latency, fine-grained coordination scheme across accelerators to dispatch and execute requests in the schedulable window. Through extensive scheduler-only benchmarks, we demonstrate that Symphony can schedule millions of requests per second and coordinate thousands of GPUs while also enabling robust autoscaling that adapts to workload changes. Symphony outperforms prior systems by achieving 5x higher goodput when given the same number of GPUs and 60% reduction in GPUs when given the same workload.