On the Expressive Power of Deep Neural Networks

TL;DR

This paper introduces a new framework for understanding neural network expressivity using trajectory length, revealing exponential complexity growth with depth, importance of initial layers, and a regularization method comparable to batch normalization.

Contribution

It presents a novel measure called trajectory length to analyze neural network expressivity and demonstrates its implications for depth complexity, layer sensitivity, and regularization techniques.

Findings

Function complexity grows exponentially with depth.

Trained networks are more sensitive to initial layer weights.

Trajectory regularization matches batch normalization performance.

Abstract

We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings can be summarized as follows: (1) The complexity of the computed function grows exponentially with depth. (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights. (3) Regularizing on trajectory length (trajectory regularization) is a simpler alternative to batch normalization, with the same performance.