A Selective Overview of Deep Learning

TL;DR

This paper provides an overview of deep learning, discussing models, training techniques, new characteristics, and recent theoretical insights, aiming to stimulate further statistical research despite incomplete understanding.

Contribution

It offers a comprehensive survey of deep learning models and techniques from a statistical perspective, highlighting new characteristics and recent theoretical developments.

Findings

Deep learning models include CNNs, RNNs, GANs.

Training techniques like SGD, dropout, batch normalization are key.

Recent theories suggest depth and over-parametrization are beneficial.

Abstract

Deep learning has arguably achieved tremendous success in recent years. In simple words, deep learning uses the composition of many nonlinear functions to model the complex dependency between input features and labels. While neural networks have a long history, recent advances have greatly improved their performance in computer vision, natural language processing, etc. From the statistical and scientific perspective, it is natural to ask: What is deep learning? What are the new characteristics of deep learning, compared with classical methods? What are the theoretical foundations of deep learning? To answer these questions, we introduce common neural network models (e.g., convolutional neural nets, recurrent neural nets, generative adversarial nets) and training techniques (e.g., stochastic gradient descent, dropout, batch normalization) from a statistical point of view. Along the way, we highlight new characteristics of deep learning (including depth and over-parametrization) and explain their practical and theoretical benefits. We also sample recent results on theories of deep learning, many of which are only suggestive. While a complete understanding of deep learning remains elusive, we hope that our perspectives and discussions serve as a stimulus for new statistical research.