The phase diagram of approximation rates for deep neural networks

TL;DR

This paper investigates the approximation capabilities of deep neural networks, revealing phase transitions in their approximation rates, and demonstrates how different architectures and activations influence these rates.

Contribution

It generalizes phase diagram results to various smoothness classes, compares activation functions, and introduces nearly exponential approximation rates for deep Fourier networks.

Findings

Deep discontinuous phase exists for arbitrary smoothness classes.

Networks with piecewise polynomial activations share the same phase diagram.

Deep Fourier networks achieve nearly exponential approximation rates.

Abstract

We explore the phase diagram of approximation rates for deep neural networks and prove several new theoretical results. In particular, we generalize the existing result on the existence of deep discontinuous phase in ReLU networks to functional classes of arbitrary positive smoothness, and identify the boundary between the feasible and infeasible rates. Moreover, we show that all networks with a piecewise polynomial activation function have the same phase diagram. Next, we demonstrate that standard fully-connected architectures with a fixed width independent of smoothness can adapt to smoothness and achieve almost optimal rates. Finally, we consider deep networks with periodic activations ("deep Fourier expansion") and prove that they have very fast, nearly exponential approximation rates, thanks to the emerging capability of the network to implement efficient lookup operations.