Characterizing Concurrency Mechanisms for NVIDIA GPUs under Deep Learning Workloads

TL;DR

This paper analyzes NVIDIA Ampere GPU concurrency mechanisms under deep learning workloads, revealing limitations in preemption, prioritization, and thread placement that hinder performance consistency and resource utilization.

Contribution

It provides a microarchitectural analysis of GPU concurrency features during deep learning tasks, highlighting key limitations affecting performance.

Findings

Lack of fine-grained preemption hampers workload scheduling

Resource contention and thread placement issues reduce GPU utilization

Deep learning workload variability challenges current GPU concurrency mechanisms

Abstract

We investigate the performance of the concurrency mechanisms available on NVIDIA's new Ampere GPU microarchitecture under deep learning training and inference workloads. In contrast to previous studies that treat the GPU as a black box, we examine scheduling at the microarchitectural level. We find that the lack of fine-grained preemption mechanisms, robust task prioritization options, and contention-aware thread block placement policies limits the effectiveness of NVIDIA's concurrency mechanisms. In summary, the sequential nature of deep learning workloads and their fluctuating resource requirements and kernel runtimes make executing such workloads while maintaining consistently high utilization and low, predictable turnaround times difficult on current NVIDIA hardware.