On the Benefits of Anticipating Load Imbalance for Performance Optimization of Parallel Applications

TL;DR

This paper proposes a novel load balancing approach for parallel applications that anticipates future load imbalances, leading to up to 16% performance improvements over standard methods.

Contribution

It introduces a new load balancing philosophy that predicts and unloads future overloaded processing elements to improve performance.

Findings

Up to 16% performance improvement observed.

Theoretical model supports the anticipatory load balancing approach.

Application to fluid model simulation validates benefits.

Abstract

In parallel iterative applications, computational efficiency is essential for addressing large problems. Load imbalance is one of the major performance degradation factors of parallel applications. Therefore, distributing, cleverly, and as evenly as possible, the workload among processing elements (PE) maximizes application performance. So far, the standard load balancing method consists in distributing the workload evenly between PEs and, when load imbalance appears, redistributing the extra load from overloaded PEs to underloaded PEs. However, this does not anticipate the load imbalance growth that may continue during the next iterations. In this paper, we present a first step toward a novel philosophy of load balancing that unloads the PEs that will be overloaded in the near future to let the application rebalance itself via its own dynamics. Herein, we present a formal definition of our new approach using a simple mathematical model and discuss its advantages compared to the standard load balancing method. In addition to the theoretical study, we apply our method to an application that reproduces the computation of a fluid model with non-uniform erosion. The performance validates the benefit of anticipating load imbalance. We observed up to 16% performance improvement compared to the standard load balancing method.