Deep Learning in Neural Networks: An Overview

TL;DR

This paper provides a comprehensive overview of deep learning, covering its history, key techniques, and various learning paradigms, highlighting its success in pattern recognition and machine learning competitions.

Contribution

It offers a historical survey of deep learning methods, clarifies the distinction between shallow and deep learners, and reviews major learning approaches including supervised, unsupervised, and reinforcement learning.

Findings

Deep neural networks have achieved notable success in pattern recognition.

The survey clarifies the concept of depth in neural networks and its significance.

Historical context of deep learning methods is summarized from early developments.

Abstract

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.