A high-level characterisation and generalisation of communication-avoiding programming techniques

TL;DR

This paper characterizes and generalizes communication-avoiding programming techniques, expanding their scope beyond traditional numerical linear algebra to better address data movement challenges on modern hardware.

Contribution

It introduces a new terminology and abstraction for communication-avoiding algorithms, broadening their application and providing a richer framework for future hardware-aware programming.

Findings

Classifies communication-avoiding algorithms with new terminology

Shows the importance of broadening the scope beyond linear algebra

Provides a generalized framework for data movement optimization

Abstract

Today's hardware's explosion of concurrency plus the explosion of data we build upon in both machine learning and scientific simulations have multifaceted impact on how we write our codes. They have changed our notion of performance and, hence, of what a good code is: Good code has, first of all, to be able to exploit the unprecedented levels of parallelism. To do so, it has to manage to move the compute data into the compute facilities on time. As communication and memory bandwidth cannot keep pace with the growth in compute capabilities and as latency increases---at least relative to what the hardware could do---communication-avoiding techniques gain importance. We characterise and classify the field of communication-avoiding algorithms. A review of some examples of communication-avoiding programming by means of our new terminology shows that we are well-advised to broaden our notion of "communication-avoiding" and to look beyond numerical linear algebra. An abstraction, generalisation and weakening of the term enriches our toolset of how to tackle the data movement challenges. Through this, we eventually gain access to a richer set of tools that we can use to deliver proper code for current and upcoming hardware generations.