The Computational Limits of Deep Learning

TL;DR

This paper examines the increasing computational demands of deep learning, highlighting its unsustainable growth and emphasizing the need for more efficient methods to sustain future progress.

Contribution

It provides a comprehensive analysis of deep learning's reliance on computational power and discusses the implications for future research directions.

Findings

Deep learning progress is strongly tied to increased computing power.

Current growth in computational requirements is unsustainable environmentally and economically.

More efficient algorithms are necessary for future advancements.

Abstract

Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks. But this progress has come with a voracious appetite for computing power. This article catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power. Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.