Nixie: Efficient, Transparent Temporal Multiplexing for Consumer GPUs

TL;DR

Nixie is a system that improves GPU memory utilization and responsiveness for consumer GPUs running large ML workloads by enabling efficient temporal multiplexing without requiring application or driver modifications.

Contribution

Nixie introduces a novel system for transparent temporal multiplexing on consumer GPUs, optimizing memory and latency without application changes.

Findings

Up to 3.8x latency improvement for interactive tasks

Reduces CPU pinned memory usage by 66.8%

Enhances GPU resource utilization efficiency

Abstract

Consumer machines are increasingly running large ML workloads such as large language models (LLMs), text-to-image generation, and interactive image editing. Unlike datacenter GPUs, consumer GPUs serve single-user, rapidly changing workloads, and each model's working set often nearly fills the GPU memory. As a result, existing sharing mechanisms (e.g., NVIDIA Unified Virtual Memory) perform poorly due to memory thrashing and excessive use of CPU pinned memory when multiple applications are active. We design and implement Nixie, a system that enables efficient and transparent temporal multiplexing on consumer GPUs without requiring any application or driver changes. Nixie is a system service that coordinates GPU memory allocation and kernel launch behavior to efficiently utilize the CPU-GPU bi-directional bandwidth and CPU pinned memory. A lightweight scheduler in Nixie further improves responsiveness by automatically prioritizing latency-sensitive interactive jobs using MLFQ-inspired techniques. Our evaluations show that Nixie improves latency of real interactive code-completion tasks by up to $3.8\times$ and saves up to 66.8% CPU pinned memory usage given the same latency requirement.