Control Flow Management in Modern GPUs

TL;DR

This paper introduces Hanoi, a low-cost control flow management mechanism for GPUs that closely mimics real hardware, enabling better performance modeling and software control despite vendor opacity.

Contribution

It defines a semantic for control flow instructions in the Turing ISA and proposes Hanoi, a mechanism that accurately replicates hardware control flow with minimal discrepancy.

Findings

Discrepancy between real hardware and Hanoi is only 1.03%

Hanoi's IPC difference from real hardware is just 0.19%

Hanoi provides a cost-effective way to model GPU control flow

Abstract

In GPUs, the control flow management mechanism determines which threads in a warp are active at any point in time. This mechanism monitors the control flow of scalar threads within a warp to optimize thread scheduling and plays a critical role in the utilization of execution resources. The control flow management mechanism can be controlled or assisted by software through instructions. However, GPU vendors do not disclose details about their compiler, ISA, or hardware implementations. This lack of transparency makes it challenging for researchers to understand how the control flow management mechanism functions, is implemented, or is assisted by software, which is crucial when it significantly affects their research. It is also problematic for performance modeling of GPUs, as one can only rely on traces from real hardware for control flow and cannot model or modify the functionality of the mechanism altering it. This paper addresses this issue by defining a plausible semantic for control flow instructions in the Turing native ISA based on insights gleaned from experimental data using various benchmarks. Based on these definitions, we propose a low-cost mechanism for efficient control flow management named Hanoi. Hanoi ensures correctness and generates a control flow that is very close to real hardware. Our evaluation shows that the discrepancy between the control flow trace of real hardware and our mechanism is only 1.03% on average. Furthermore, when comparing the Instructions Per Cycle (IPC) of GPUs employing Hanoi with the native control flow management of actual hardware, the average difference is just 0.19%.