Classifying Overlapping Gaussian Mixtures in High Dimensions: From Optimal Classifiers to Neural Nets

TL;DR

This paper derives explicit formulas for optimal decision boundaries in high-dimensional Gaussian mixture classification and shows neural networks approximate these boundaries, revealing how they learn statistical patterns from complex data.

Contribution

It provides the first closed-form expressions for Bayes optimal classifiers in high-dimensional GMMs and links neural network decision thresholds to covariance eigenstructure.

Findings

Neural networks approximate optimal classifiers derived for GMMs.

Decision thresholds in neural networks relate to covariance eigenvectors.

Theoretical insights into neural networks' probabilistic inference capabilities.

Abstract

We derive closed-form expressions for the Bayes optimal decision boundaries in binary classification of high dimensional overlapping Gaussian mixture model (GMM) data, and show how they depend on the eigenstructure of the class covariances, for particularly interesting structured data. We empirically demonstrate, through experiments on synthetic GMMs inspired by real-world data, that deep neural networks trained for classification, learn predictors which approximate the derived optimal classifiers. We further extend our study to networks trained on authentic data, observing that decision thresholds correlate with the covariance eigenvectors rather than the eigenvalues, mirroring our GMM analysis. This provides theoretical insights regarding neural networks' ability to perform probabilistic inference and distill statistical patterns from intricate distributions.