Representation Benefits of Deep Feedforward Networks

TL;DR

This paper demonstrates that deep feedforward networks with ReLU nonlinearities can efficiently solve certain classification problems that shallow networks cannot, highlighting the representational advantages of depth.

Contribution

It introduces a family of classification problems where deep networks succeed while shallow networks with fewer nodes fail, emphasizing the importance of depth in neural network design.

Findings

Deep networks with 2 nodes per layer achieve zero error on the problem.

Shallow networks with fewer than exponentially many nodes have at least 1/6 error.

Recurrent networks with 3 nodes iterated k times also achieve zero error.

Abstract

This note provides a family of classification problems, indexed by a positive integer $k$, where all shallow networks with fewer than exponentially (in $k$) many nodes exhibit error at least $1/6$, whereas a deep network with 2 nodes in each of $2k$ layers achieves zero error, as does a recurrent network with 3 distinct nodes iterated $k$ times. The proof is elementary, and the networks are standard feedforward networks with ReLU (Rectified Linear Unit) nonlinearities.