Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units

TL;DR

This paper extends theoretical understanding of narrow deep belief networks with discrete units, showing how their depth and width relate to their ability to approximate any probability distribution within a specified error tolerance.

Contribution

It generalizes previous binary-unit results to arbitrary finite state units and quantifies the depth needed for universal approximation with controlled error.

Findings

Deep belief networks with specified layers can approximate any distribution within a KL divergence threshold.

The analysis applies to discrete RBMs and naive Bayes models as special cases.

Provides bounds on network depth and width for universal approximation.

Abstract

We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever and Hinton, 2008; Le Roux and Bengio, 2008, 2010; Mont\'ufar and Ay, 2011) to units with arbitrary finite state spaces, and the vanishing approximation error to an arbitrary approximation error tolerance. For example, we show that a $q$-ary deep belief network with $L\geq 2+\frac{q^{\lceil m-\delta \rceil}-1}{q-1}$ layers of width $n \leq m + \log_q(m) + 1$ for some $m\in \mathbb{N}$ can approximate any probability distribution on $\{0,1,\ldots,q-1\}^n$ without exceeding a Kullback-Leibler divergence of $\delta$. Our analysis covers discrete restricted Boltzmann machines and na\"ive Bayes models as special cases.