Statistical model-based evaluation of neural networks

TL;DR

This paper introduces a statistical model-based framework for evaluating neural networks, enabling benchmarking against MMSE bounds and analyzing effects of data characteristics on performance.

Contribution

It presents a novel experimental setup using Gaussian mixture models to systematically assess neural networks under various data conditions.

Findings

Neural network performance varies significantly with data size and geometry.

Benchmarking against MMSE bounds reveals the potential and limitations of NNs.

Understanding data statistics is crucial for optimal neural network design.

Abstract

Using a statistical model-based data generation, we develop an experimental setup for the evaluation of neural networks (NNs). The setup helps to benchmark a set of NNs vis-a-vis minimum-mean-square-error (MMSE) performance bounds. This allows us to test the effects of training data size, data dimension, data geometry, noise, and mismatch between training and testing conditions. In the proposed setup, we use a Gaussian mixture distribution to generate data for training and testing a set of competing NNs. Our experiments show the importance of understanding the type and statistical conditions of data for appropriate application and design of NNs