Communication Lower Bounds and Optimal Algorithms for Symmetric Matrix Computations

TL;DR

This paper establishes tight communication lower bounds and presents optimal algorithms for symmetric matrix computations like SYRK, SYR2K, and SYMM, crucial in linear algebra applications, using geometric and optimization techniques.

Contribution

It provides the first tight communication bounds for these symmetric matrix operations and designs algorithms that achieve these bounds in both sequential and parallel models.

Findings

Derived tight communication lower bounds for SYRK, SYR2K, and SYMM.

Developed communication-optimal algorithms matching the bounds.

Applied geometric inequalities and nonlinear optimization in proofs.

Abstract

In this article, we focus on the communication costs of three symmetric matrix computations: i) multiplying a matrix with its transpose, known as a symmetric rank-k update (SYRK) ii) adding the result of the multiplication of a matrix with the transpose of another matrix and the transpose of that result, known as a symmetric rank-2k update (SYR2K) iii) performing matrix multiplication with a symmetric input matrix (SYMM). All three computations appear in the Level 3 Basic Linear Algebra Subroutines (BLAS) and have wide use in applications involving symmetric matrices. We establish communication lower bounds for these kernels using sequential and distributed-memory parallel computational models, and we show that our bounds are tight by presenting communication-optimal algorithms for each setting. Our lower bound proofs rely on applying a geometric inequality for symmetric computations and analytically solving constrained nonlinear optimization problems. The symmetric matrix and its corresponding computations are accessed and performed according to a triangular block partitioning scheme in the optimal algorithms.