ILU Smoothers for Low Mach Navier-Stokes Pressure Solvers

TL;DR

This paper introduces a novel approach using row scaling and Richardson iteration to replace slow direct triangular solves in ILU smoothers, significantly improving parallel GPU performance in pressure solvers.

Contribution

It proposes a scalable, GPU-friendly ILU smoother with iterative triangular solves, enhancing parallel efficiency and reducing solution time in large-scale fluid dynamics simulations.

Findings

Iterative triangular solves outperform direct solves on GPUs.

Scaling reduces matrix departure from normality, improving convergence.

GMRES+AMG is at least five times faster with iterative solves.

Abstract

Incomplete LU (ILU) smoothers are effective in the algebraic multigrid (AMG) $V$-cycle for reducing high-frequency components of the error. However, the requisite direct triangular solves are comparatively slow on GPUs. Previous work has demonstrated the advantages of Jacobi iteration as an alternative to direct solution of these systems. Depending on the threshold and fill-level parameters chosen, the factors can be highly non-normal and Jacobi is unlikely to converge in a low number of iterations. We demonstrate that row scaling can reduce the departure from normality, allowing us to replace the inherently sequential solve with a rapidly converging Richardson iteration. There are several advantages beyond the lower compute time. Scaling is performed locally for a diagonal block of the global matrix because it is applied directly to the factor. Further, an ILUT Schur complement smoother maintains a constant GMRES iteration count as the number of MPI ranks increases, and thus parallel strong-scaling is improved. Our algorithms have been incorporated into hypre, and we demonstrate improved time to solution for linear systems arising in the Nalu-Wind and PeleLM pressure solvers. For large problem sizes, GMRES$+$AMG executes at least five times faster when using iterative triangular solves compared with direct solves on massively-parallel GPUs.