Massively Parallel Algorithms for the Lattice Boltzmann Method on Non-uniform Grids

TL;DR

This paper introduces parallel algorithms and data structures for the lattice Boltzmann method on non-uniform grids, enabling highly scalable large-scale flow simulations on supercomputers with near-trillion cell updates per second.

Contribution

It presents novel parallel algorithms and data structures supporting large-scale, non-uniform grid lattice Boltzmann simulations, implemented in the waLBerla framework.

Findings

Near-perfect scalability on Blue Gene/Q with almost two million threads.

Achieved close to a trillion lattice Boltzmann cell updates per second.

Simulation of over 8.5 million cells with less than one millisecond per time step.

Abstract

The lattice Boltzmann method exhibits excellent scalability on current supercomputing systems and has thus increasingly become an alternative method for large-scale non-stationary flow simulations, reaching up to a trillion grid nodes. Additionally, grid refinement can lead to substantial savings in memory and compute time. These saving, however, come at the cost of much more complex data structures and algorithms. In particular, the interface between subdomains with different grid sizes must receive special treatment. In this article, we present parallel algorithms, distributed data structures, and communication routines that are implemented in the software framework waLBerla in order to support large-scale, massively parallel lattice Boltzmann-based simulations on non-uniform grids. Additionally, we evaluate the performance of our approach on two current petascale supercomputers. On an IBM Blue Gene/Q system, the largest weak scaling benchmarks with refined grids are executed with almost two million threads, demonstrating not only near-perfect scalability but also an absolute performance of close to a trillion lattice Boltzmann cell updates per second. On an Intel-based system, the strong scaling of a simulation with refined grids and a total of more than 8.5 million cells is demonstrated to reach a performance of less than one millisecond per time step. This enables simulations with complex, non-uniform grids and four million time steps per hour compute time.