The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design

TL;DR

This paper reviews recent advances in deep learning and explores their implications for future computer architecture, chip design, and potential for more dynamic, scalable models beyond current capabilities.

Contribution

It discusses how deep learning advances influence hardware design, circuit development, and proposes directions for larger, more flexible multi-task models.

Findings

Deep learning has significantly impacted AI capabilities across multiple domains.

Implications for post-Moore's Law hardware development are examined.

Potential for large-scale, sparsely activated multi-task models is highlighted.

Abstract

The past decade has seen a remarkable series of advances in machine learning, and in particular deep learning approaches based on artificial neural networks, to improve our abilities to build more accurate systems across a broad range of areas, including computer vision, speech recognition, language translation, and natural language understanding tasks. This paper is a companion paper to a keynote talk at the 2020 International Solid-State Circuits Conference (ISSCC) discussing some of the advances in machine learning, and their implications on the kinds of computational devices we need to build, especially in the post-Moore's Law-era. It also discusses some of the ways that machine learning may also be able to help with some aspects of the circuit design process. Finally, it provides a sketch of at least one interesting direction towards much larger-scale multi-task models that are sparsely activated and employ much more dynamic, example- and task-based routing than the machine learning models of today.