Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities

TL;DR

This paper provides a comprehensive overview of algorithmically-efficient deep learning, formalizing the problem, developing a taxonomy of methods, and discussing evaluation practices, bottlenecks, and future research directions to reduce training costs.

Contribution

It offers a structured taxonomy of algorithmic speedup methods, formalizes the problem, and discusses evaluation and bottleneck mitigation strategies in deep learning training.

Findings

Identification of commonalities among diverse speedup methods

Development of evaluation best practices for fair comparison

Analysis of training pipeline bottlenecks and mitigation strategies

Abstract

Although deep learning has made great progress in recent years, the exploding economic and environmental costs of training neural networks are becoming unsustainable. To address this problem, there has been a great deal of research on *algorithmically-efficient deep learning*, which seeks to reduce training costs not at the hardware or implementation level, but through changes in the semantics of the training program. In this paper, we present a structured and comprehensive overview of the research in this field. First, we formalize the *algorithmic speedup* problem, then we use fundamental building blocks of algorithmically efficient training to develop a taxonomy. Our taxonomy highlights commonalities of seemingly disparate methods and reveals current research gaps. Next, we present evaluation best practices to enable comprehensive, fair, and reliable comparisons of speedup techniques. To further aid research and applications, we discuss common bottlenecks in the training pipeline (illustrated via experiments) and offer taxonomic mitigation strategies for them. Finally, we highlight some unsolved research challenges and present promising future directions.