Performance Analysis and Optimal Node-Aware Communication for Enlarged Conjugate Gradient Methods

TL;DR

This paper analyzes the performance of Enlarged Conjugate Gradient methods for large sparse systems, highlighting communication bottlenecks and proposing node-aware communication techniques to improve scalability.

Contribution

It provides a detailed performance study of ECG, models its expected performance, and introduces a novel node-aware communication approach to enhance scalability.

Findings

Increased communication overhead due to denser messages in block vector operations.

Performance models effectively predict ECG scalability.

Node-aware communication significantly improves efficiency at scale.

Abstract

Krylov methods are a key way of solving large sparse linear systems of equations, but suffer from poor strong scalabilty on distributed memory machines. This is due to high synchronization costs from large numbers of collective communication calls alongside a low computational workload. Enlarged Krylov methods address this issue by decreasing the total iterations to convergence, an artifact of splitting the initial residual and resulting in operations on block vectors. In this paper, we present a performance study of an Enlarged Krylov Method, Enlarged Conjugate Gradients (ECG), noting the impact of block vectors on parallel performance at scale. Most notably, we observe the increased overhead of point-to-point communication as a result of denser messages in the sparse matrix-block vector multiplication kernel. Additionally, we present models to analyze expected performance of ECG, as well as, motivate design decisions. Most importantly, we introduce a new point-to-point communication approach based on node-aware communication techniques that increases efficiency of the method at scale.