MQFQ-Sticky: Fair Queueing For Serverless GPU Functions

TL;DR

This paper introduces MQFQ-Sticky, a novel GPU scheduling system for serverless functions that significantly reduces latency by applying fair queueing principles, enabling efficient GPU acceleration without modifying function code.

Contribution

It presents MQFQ-Sticky, a new GPU scheduling and memory management approach for serverless functions that improves fairness and reduces latency in heterogeneous workloads.

Findings

Reduces function latency by 2x to 20x compared to existing policies.

Balances locality, fairness, and latency effectively.

Supports GPU acceleration in serverless environments without code modifications.

Abstract

Hardware accelerators like GPUs are now ubiquitous in data centers, but are not fully supported by common cloud abstractions such as Functions as a Service (FaaS). Many popular and emerging FaaS applications such as machine learning and scientific computing can benefit from GPU acceleration. However, FaaS frameworks (such as OpenWhisk) are not capable of providing this acceleration because of the impedance mismatch between GPUs and the FaaS programming model, which requires virtualization and sandboxing of each function. The challenges are amplified due to the highly dynamic and heterogeneous FaaS workloads. This paper presents the design and implementation of a FaaS system for providing GPU acceleration in a black-box manner (without modifying function code). Running small functions in containerized sandboxes is challenging due to limited GPU concurrency and high cold-start overheads, resulting in heavy queueing of function invocations. We show how principles from I/O scheduling, such as fair queuing and anticipatory scheduling, can be translated to function scheduling on GPUs. We develop MQFQ-Sticky, an integrated fair queueing and GPU memory management approach, which balances the tradeoffs between locality, fairness, and latency. Empirical evaluation on a range of workloads shows that it reduces function latency by 2x to 20x compared to existing GPU and CPU queueing policies.