Towards QoS-Aware and Resource-Efficient GPU Microservices Based on Spatial Multitasking GPUs In Datacenters

TL;DR

This paper introduces Camelot, a runtime system for GPU microservices that optimizes resource utilization and QoS through a global communication mechanism and contention-aware resource policies, outperforming existing solutions.

Contribution

Camelot is the first system to address GPU microservice resource management considering contention and pipeline effects, improving peak load support and resource efficiency.

Findings

Supports up to 64.5% higher peak load with limited GPUs

Reduces resource usage by 35% at low load

Achieves 99th percentile latency targets

Abstract

While prior researches focus on CPU-based microservices, they are not applicable for GPU-based microservices due to the different contention patterns. It is challenging to optimize the resource utilization while guaranteeing the QoS for GPU microservices. We find that the overhead is caused by inter microservice communication, GPU resource contention and imbalanced throughput within microservice pipeline. We propose Camelot, a runtime system that manages GPU micorservices considering the above factors. In Camelot, a global memory-based communication mechanism enables onsite data sharing that significantly reduces the end-to-end latencies of user queries. We also propose two contention aware resource allocation policies that either maximize the peak supported service load or minimize the resource usage at low load while ensuring the required QoS. The two policies consider the microservice pipeline effect and the runtime GPU resource contention when allocating resources for the microservices. Compared with state-of-the-art work, Camelot increases the supported peak load by up to 64.5% with limited GPUs, and reduces 35% resource usage at low load while achieving the desired 99%-ile latency target.