Orthogonal layers of parallelism in large-scale eigenvalue computations

TL;DR

This paper proposes a novel approach to improve scalability in large-scale eigenvalue computations by introducing two orthogonal layers of parallelism, reducing communication overhead in distributed sparse matrix operations.

Contribution

It introduces a new parallelization strategy with orthogonal layers that significantly enhances scalability in distributed eigenvalue computations, validated through theoretical analysis and benchmarks.

Findings

Scalability can be restored with two orthogonal parallelism layers.

The approach reduces communication overhead in large-scale eigenvalue problems.

Benchmarks confirm improved performance in physics and optimization applications.

Abstract

We address the communication overhead of distributed sparse matrix-(multiple)-vector multiplication in the context of large-scale eigensolvers, using filter diagonalization as an example. The basis of our study is a performance model which includes a communication metric that is computed directly from the matrix sparsity pattern without running any code. The performance model quantifies to which extent scalability and parallel efficiency are lost due to communication overhead. To restore scalability, we identify two orthogonal layers of parallelism in the filter diagonalization technique. In the horizontal layer the rows of the sparse matrix are distributed across individual processes. In the vertical layer bundles of multiple vectors are distributed across separate process groups. An analysis in terms of the communication metric predicts that scalability can be restored if, and only if, one implements the two orthogonal layers of parallelism via different distributed vector layouts. Our theoretical analysis is corroborated by benchmarks for application matrices from quantum and solid state physics, road networks, and nonlinear programming. We finally demonstrate the benefits of using orthogonal layers of parallelism with two exemplary application cases -- an exciton and a strongly correlated electron system -- which incur either small or large communication overhead.