Efficient Deep Learning of GMMs

TL;DR

This paper demonstrates that deep neural networks can efficiently classify Gaussian mixture models in high-dimensional spaces using linear neuron counts, unlike shallow networks which require exponentially more resources.

Contribution

It proves that deep neural networks can classify GMMs in high dimensions with linear complexity, highlighting the efficiency of depth in neural network architectures.

Findings

Deep networks classify GMMs with O(n) neurons.

Shallow networks need exponential neurons for the same task.

Results explain the practical success of deep learning in high-dimensional data.

Abstract

We show that a collection of Gaussian mixture models (GMMs) in $R^{n}$ can be optimally classified using $O(n)$ neurons in a neural network with two hidden layers (deep neural network), whereas in contrast, a neural network with a single hidden layer (shallow neural network) would require at least $O(\exp(n))$ neurons or possibly exponentially large coefficients. Given the universality of the Gaussian distribution in the feature spaces of data, e.g., in speech, image and text, our result sheds light on the observed efficiency of deep neural networks in practical classification problems.