Refinements of Universal Approximation Results for Deep Belief Networks and Restricted Boltzmann Machines

TL;DR

This paper refines the understanding of the resources needed by RBMs and DBNs to serve as universal approximators, providing tighter bounds and confirming a previous conjecture about their capabilities.

Contribution

It improves existing bounds on the number of hidden units and layers required for RBMs and DBNs to approximate any distribution on binary vectors.

Findings

RBMs with k-1 hidden units can approximate any distribution, where k is minimal based on support set union.

Constructed a DBN with 2^{n/2}(n - log(n)) hidden units per layer that can approximate any distribution.

Confirmed a conjecture by Le Roux and Bengio (2010) regarding DBN approximation capabilities.

Abstract

We improve recently published results about resources of Restricted Boltzmann Machines (RBM) and Deep Belief Networks (DBN) required to make them Universal Approximators. We show that any distribution p on the set of binary vectors of length n can be arbitrarily well approximated by an RBM with k-1 hidden units, where k is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of p. In important cases this number is half of the cardinality of the support set of p. We construct a DBN with 2^n/2(n-b), b ~ log(n), hidden layers of width n that is capable of approximating any distribution on {0,1}^n arbitrarily well. This confirms a conjecture presented by Le Roux and Bengio 2010.