Two-level Dynamic Load Balancing for High Performance Scientific Applications

TL;DR

This paper proposes a two-level dynamic load balancing approach for high-performance scientific applications on HPC systems, demonstrating up to 21% performance improvement by addressing load imbalance at both thread and process levels.

Contribution

It introduces a generic, joint load balancing method using extended LaPeSD and DLS4LB libraries for MPI and OpenMP applications, addressing multi-level load imbalance.

Findings

Joint load balancing improves performance by up to 21%.

Different DLS techniques have varying effectiveness at each level.

Addressing both levels is essential for optimal performance.

Abstract

Scientific applications are often complex, irregular, and computationally-intensive. To accommodate the ever-increasing computational demands of scientific applications, high-performance computing (HPC) systems have become larger and more complex, offering parallelism at multiple levels (e.g., nodes, cores per node, threads per core). Scientific applications need to exploit all the available multilevel hardware parallelism to harness the available computational power. The performance of applications executing on such HPC systems may adversely be affected by load imbalance at multiple levels, caused by problem, algorithmic, and systemic characteristics. Nevertheless, most existing load balancing methods do not simultaneously address load imbalance at multiple levels. This work investigates the impact of load imbalance on the performance of three scientific applications at the thread and process levels. We jointly apply and evaluate selected dynamic loop self-scheduling (DLS) techniques to both levels. Specifically, we employ the extended LaPeSD OpenMP runtime library at the thread level and extend the DLS4LB MPI-based dynamic load balancing library at the process level. This approach is generic and applicable to any multiprocess-multithreaded computationally-intensive application (programmed using MPI and OpenMP). We conduct an exhaustive set of experiments to assess and compare six DLS techniques at the thread level and eleven at the process level. The results show that improved application performance, by up to 21%, can only be achieved by jointly addressing load imbalance at the two levels. We offer insights into the performance of the selected DLS techniques and discuss the interplay of load balancing at the thread level and process level.