On the Origin of Deep Learning

TL;DR

This paper reviews the historical development of deep learning models, tracing their origins from early neural networks to modern architectures, and discusses how past ideas shaped current models and future research directions.

Contribution

It provides a comprehensive overview of the evolutionary history of deep learning models, highlighting the development paths and foundational ideas behind major architectures.

Findings

Deep learning models have diverse evolutionary paths spanning over half a century.

Many modern models are inspired by biological vision and ancient linear models.

The paper offers guidance for future research directions in deep learning.

Abstract

This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.