A Test for FLOPs as a Discriminant for Linear Algebra Algorithms

TL;DR

This paper proposes a methodology to evaluate whether FLOPs can reliably discriminate between linear algebra algorithms based on their performance, highlighting cases where FLOPs are insufficient for optimal algorithm selection.

Contribution

The work introduces a ranking-based methodology to test FLOPs as a performance discriminant for linear algebra algorithms, identifying anomalies where FLOPs do not predict execution time.

Findings

FLOPs are sometimes unreliable as a performance indicator.

The methodology can classify algorithms into performance classes.

Anomalies reveal cases where FLOPs do not correlate with execution time.

Abstract

Linear algebra expressions, which play a central role in countless scientific computations, are often computed via a sequence of calls to existing libraries of building blocks (such as those provided by BLAS and LAPACK). A sequence identifies a computing strategy, i.e., an algorithm, and normally for one linear algebra expression many alternative algorithms exist. Although mathematically equivalent, those algorithms might exhibit significant differences in terms of performance. Several high-level languages and tools for matrix computations such as Julia, Armadillo, Linnea, etc., make algorithmic choices by minimizing the number of Floating Point Operations (FLOPs). However, there can be several algorithms that share the same (or have nearly identical) number of FLOPs; in many cases, these algorithms exhibit execution times which are statistically equivalent and one could arbitrarily select one of them as the best algorithm. It is however not unlikely to find cases where the execution times are significantly different from one another (despite the FLOP count being almost the same). It is also possible that the algorithm that minimizes FLOPs is not the one that minimizes execution time. In this work, we develop a methodology to test the reliability of FLOPs as discriminant for linear algebra algorithms. Given a set of algorithms (for an instance of a linear algebra expression) as input, the methodology ranks them into performance classes; i.e., multiple algorithms are allowed to share the same rank. To this end, we measure the algorithms iteratively until the changes in the ranks converge to a value close to zero. FLOPs are a valid discriminant for an instance if all the algorithms with minimum FLOPs are assigned the best rank; otherwise, the instance is regarded as an anomaly, which can then be used in the investigation of the root cause of performance differences.