Quantitative Universal Approximation Bounds for Deep Belief Networks

TL;DR

This paper establishes that deep belief networks with binary hidden units can universally approximate any multivariate probability density under mild conditions, providing explicit bounds on the approximation error based on network size.

Contribution

It provides the first quantitative bounds on the approximation capabilities of deep belief networks in terms of the number of hidden units.

Findings

Deep belief networks can approximate any multivariate density under mild conditions.

Explicit bounds on approximation error are derived in terms of hidden units.

The approximation is valid in both $L^q$-norm and Kullback-Leibler divergence.

Abstract

We show that deep belief networks with binary hidden units can approximate any multivariate probability density under very mild integrability requirements on the parental density of the visible nodes. The approximation is measured in the $L^q$-norm for $q\in[1,\infty]$ ($q=\infty$ corresponding to the supremum norm) and in Kullback-Leibler divergence. Furthermore, we establish sharp quantitative bounds on the approximation error in terms of the number of hidden units.