Generalizing Hierarchical Parallelism

TL;DR

This paper proposes a simplified, nested parallelism model for OpenMP to better handle the increasing complexity of hardware hierarchies, aiming for more flexible and scalable parallel programming.

Contribution

It introduces a generalized hierarchical parallelism approach that revisits nested parallelism, simplifying the model and improving flexibility for deep hardware hierarchies.

Findings

Enhanced flexibility in handling hardware hierarchies

Simplified model for nested parallelism

Potential for improved scalability and programmability

Abstract

Since the days of OpenMP 1.0 computer hardware has become more complex, typically by specializing compute units for coarse- and fine-grained parallelism in incrementally deeper hierarchies of parallelism. Newer versions of OpenMP reacted by introducing new mechanisms for querying or controlling its individual levels, each time adding another concept such as places, teams, and progress groups. In this paper we propose going back to the roots of OpenMP in the form of nested parallelism for a simpler model and more flexible handling of arbitrary deep hardware hierarchies.