Approximating Probability Distributions by ReLU Networks

TL;DR

This paper investigates the number of neurons needed in ReLU neural networks to approximate histogram probability distributions, providing new bounds that improve upon previous results.

Contribution

It introduces a novel upper bound on neuron count for approximation and establishes a lower bound, advancing understanding of neural network approximation capabilities.

Findings

New upper bound on neurons needed for approximation

Improved bounds over previous results

Efficient construction of neural nets for piecewise linear functions

Abstract

How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.