Towards automated generation of fast and accurate algorithms for recursive matrix multiplication

TL;DR

This paper introduces a novel strategy for automatically generating fast, accurate recursive matrix multiplication algorithms by optimizing stability bounds, operation count, and basis sparsification, exemplified by improved 2x2 matrix multiplication algorithms.

Contribution

It presents a comprehensive, three-phase strategy for designing recursive matrix multiplication algorithms that optimize stability, efficiency, and accuracy, including a new 2x2 algorithm with better practical accuracy.

Findings

Proposed a 2x2 matrix multiplication algorithm with 7 products outperforming previous methods.

Developed heuristics that reduce operation count while maintaining accuracy.

Applied basis sparsification to improve time complexity constants.

Abstract

We propose a strategy for the generation of fast and accurate versions of non-commutative recursive matrix multiplication algorithms. To generate these algorithms, we consider matrix and tensor norm bounds governing the stability and accuracy of numerical matrix multiplication. We start by a unification on known max-norm bounds on matrix multiplication stability and then extend them to further norms and more generally to recursive bilinear algorithms and the alternative basis matrix multiplication algorithms. Then our strategy has three phases. First, we reduce those bounds by minimizing a growth factor along the orbits of the associated matrix multiplication tensor decomposition. Second, we develop heuristics that minimize the number of operations required to realize a bilinear formula, while further improving its accuracy. Third, we perform an alternative basis sparsification that improves on the time complexity constant and mostly preserves the overall accuracy. For instance this strategy allows us to propose a non-commutative algorithm for multiplying 2x2-matrices using 7 coefficient products. This algorithm reaches simultaneously a better accuracy in practice compared to previously known such fast ___2x2x2:7___ Strassen-like algorithms and a time complexity bound with the best currently known leading term (obtained via alternative basis sparsification). We also present detailed results of our technique on other recursive matrix multiplication algorithms, such as Smirnov's ___3x3x6:40___ family of algorithms.