Weighted decomposition in high-performance lattice-Boltzmann simulations: are some lattice sites more equal than others?

TL;DR

This paper investigates weighted domain decomposition and site sorting to improve load balancing in high-performance lattice-Boltzmann simulations of sparse geometries, achieving significant reduction in load imbalance.

Contribution

It introduces a weighted decomposition approach combined with space filling curve sorting to enhance load balance in sparse lattice-Boltzmann simulations.

Findings

Weighted decomposition reduces load imbalance by up to 85%.

The approach increases communication overhead in some cases.

Effective for complex sparse geometries like aneurysms.

Abstract

Obtaining a good load balance is a significant challenge in scaling up lattice-Boltzmann simulations of realistic sparse problems to the exascale. Here we analyze the effect of weighted decomposition on the performance of the HemeLB lattice-Boltzmann simulation environment, when applied to sparse domains. Prior to domain decomposition, we assign wall and in/outlet sites with increased weights which reflect their increased computational cost. We combine our weighted decomposition with a second optimization, which is to sort the lattice sites according to a space filling curve. We tested these strategies on a sparse bifurcation and very sparse aneurysm geometry, and find that using weights reduces calculation load imbalance by up to 85%, although the overall communication overhead is higher than some of our runs.