Improving Multi-Application Concurrency Support Within the GPU Memory System

TL;DR

This paper identifies the memory system as a key bottleneck in multi-application GPU execution and proposes MASK, a new memory hierarchy design that improves virtual memory support and reduces contention.

Contribution

The paper introduces MASK, a novel GPU memory hierarchy extension that enhances multi-application concurrency by reducing TLB contention and improving address translation efficiency.

Findings

MASK reduces TLB miss rates significantly.

Improves GPU throughput during multi-application workloads.

Decreases inter-core thrashing in GPU memory system.

Abstract

GPUs exploit a high degree of thread-level parallelism to hide long-latency stalls. Due to the heterogeneous compute requirements of different applications, there is a growing need to share the GPU across multiple applications in large-scale computing environments. However, while CPUs offer relatively seamless multi-application concurrency, and are an excellent fit for multitasking and for virtualized environments, GPUs currently offer only primitive support for multi-application concurrency. Much of the problem in a contemporary GPU lies within the memory system, where multi-application execution requires virtual memory support to manage the address spaces of each application and to provide memory protection. In this work, we perform a detailed analysis of the major problems in state-of-the-art GPU virtual memory management that hinders multi-application execution. Existing GPUs are designed to share memory between the CPU and GPU, but do not handle multi-application support within the GPU well. We find that when multiple applications spatially share the GPU, there is a significant amount of inter-core thrashing on the shared TLB within the GPU. The TLB contention is high enough to prevent the GPU from successfully hiding stall latencies, thus becoming a first-order performance concern. We introduce MASK, a memory hierarchy design that provides low-overhead virtual memory support for the concurrent execution of multiple applications. MASK extends the GPU memory hierarchy to efficiently support address translation through the use of multi-level TLBs, and uses translation-aware memory and cache management to maximize throughput in the presence of inter-application contention.