Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization

TL;DR

This paper reviews recent theoretical advances in deep learning, focusing on representation power, optimization landscapes, and generalization, especially in overparametrized networks, providing insights into their surprising effectiveness.

Contribution

It synthesizes recent theoretical results across approximation, optimization, and generalization, offering a unified understanding of deep networks' success.

Findings

Deep networks can approximate any continuous function with exponential parameters.

Deep convolutional networks can efficiently approximate compositional functions.

SGD is likely to find global minima in exponential loss landscapes.

Abstract

While deep learning is successful in a number of applications, it is not yet well understood theoretically. A satisfactory theoretical characterization of deep learning however, is beginning to emerge. It covers the following questions: 1) representation power of deep networks 2) optimization of the empirical risk 3) generalization properties of gradient descent techniques --- why the expected error does not suffer, despite the absence of explicit regularization, when the networks are overparametrized? In this review we discuss recent advances in the three areas. In approximation theory both shallow and deep networks have been shown to approximate any continuous functions on a bounded domain at the expense of an exponential number of parameters (exponential in the dimensionality of the function). However, for a subset of compositional functions, deep networks of the convolutional type can have a linear dependence on dimensionality, unlike shallow networks. In optimization we discuss the loss landscape for the exponential loss function and show that stochastic gradient descent will find with high probability the global minima. To address the question of generalization for classification tasks, we use classical uniform convergence results to justify minimizing a surrogate exponential-type loss function under a unit norm constraint on the weight matrix at each layer -- since the interesting variables for classification are the weight directions rather than the weights. Our approach, which is supported by several independent new results, offers a solution to the puzzle about generalization performance of deep overparametrized ReLU networks, uncovering the origin of the underlying hidden complexity control.