Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networks

TL;DR

This paper provides non-asymptotic bounds and convergence rates for deep ReLU networks classifying data from Gaussian Mixture Models, showing they can overcome the curse of dimensionality in unbounded domains.

Contribution

It introduces the first non-asymptotic analysis of deep ReLU networks for GMM data, with bounds independent of dimension, and develops a novel approximation error bound for analytic functions.

Findings

Deep ReLU networks achieve dimension-independent convergence rates.

The analysis applies to unbounded domains using Gaussian distribution properties.

Results support the practical efficiency of deep neural networks in classification tasks.

Abstract

This paper studies the binary classification of unbounded data from ${\mathbb R}^d$ generated under Gaussian Mixture Models (GMMs) using deep ReLU neural networks. We obtain $\unicode{x2013}$ for the first time $\unicode{x2013}$ non-asymptotic upper bounds and convergence rates of the excess risk (excess misclassification error) for the classification without restrictions on model parameters. The convergence rates we derive do not depend on dimension $d$, demonstrating that deep ReLU networks can overcome the curse of dimensionality in classification. While the majority of existing generalization analysis of classification algorithms relies on a bounded domain, we consider an unbounded domain by leveraging the analyticity and fast decay of Gaussian distributions. To facilitate our analysis, we give a novel approximation error bound for general analytic functions using ReLU networks, which may be of independent interest. Gaussian distributions can be adopted nicely to model data arising in applications, e.g., speeches, images, and texts; our results provide a theoretical verification of the observed efficiency of deep neural networks in practical classification problems.