An Infinite Restricted Boltzmann Machine

TL;DR

This paper introduces an infinite restricted Boltzmann machine (RBM) with an adaptive hidden layer that grows during training, eliminating the need to predefine the number of hidden units.

Contribution

It extends RBMs to handle an infinite number of hidden units by making the hidden layer sensitive to order and defining a limiting energy function.

Findings

Performance comparable to standard RBMs

Eliminates the need for hidden layer size tuning

Demonstrates stable training with infinite hidden units

Abstract

We present a mathematical construction for the restricted Boltzmann machine (RBM) that doesn't require specifying the number of hidden units. In fact, the hidden layer size is adaptive and can grow during training. This is obtained by first extending the RBM to be sensitive to the ordering of its hidden units. Then, thanks to a carefully chosen definition of the energy function, we show that the limit of infinitely many hidden units is well defined. As with RBM, approximate maximum likelihood training can be performed, resulting in an algorithm that naturally and adaptively adds trained hidden units during learning. We empirically study the behaviour of this infinite RBM, showing that its performance is competitive to that of the RBM, while not requiring the tuning of a hidden layer size.