Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators

TL;DR

This paper proves that deep neural network discriminators have a provable advantage over shallow ones in distinguishing certain distributions, highlighting the importance of depth and feature learning in GANs.

Contribution

The authors construct specific distribution pairs demonstrating that deep discriminators outperform shallow ones, providing theoretical evidence for the benefits of depth and feature learning.

Findings

Deep discriminators distinguish distributions better than shallow ones.

Depth enables discriminators to detect differences that shallow networks cannot.

Feature learning enhances the discriminative power of neural networks.

Abstract

We construct pairs of distributions $\mu_d, \nu_d$ on $\mathbb{R}^d$ such that the quantity $|\mathbb{E}_{x \sim \mu_d} [F(x)] - \mathbb{E}_{x \sim \nu_d} [F(x)]|$ decreases as $\Omega(1/d^2)$ for some three-layer ReLU network $F$ with polynomial width and weights, while declining exponentially in $d$ if $F$ is any two-layer network with polynomial weights. This shows that deep GAN discriminators are able to distinguish distributions that shallow discriminators cannot. Analogously, we build pairs of distributions $\mu_d, \nu_d$ on $\mathbb{R}^d$ such that $|\mathbb{E}_{x \sim \mu_d} [F(x)] - \mathbb{E}_{x \sim \nu_d} [F(x)]|$ decreases as $\Omega(1/(d\log d))$ for two-layer ReLU networks with polynomial weights, while declining exponentially for bounded-norm functions in the associated RKHS. This confirms that feature learning is beneficial for discriminators. Our bounds are based on Fourier transforms.