Learning Infinite RBMs with Frank-Wolfe

TL;DR

This paper introduces an infinite RBM model optimized via Frank-Wolfe, enabling adaptive complexity and improved training initialization, resulting in higher test likelihoods.

Contribution

It proposes a novel infinite RBM framework with Frank-Wolfe optimization, allowing automatic model complexity control and better initialization for training.

Findings

Sparse solutions with increasing hidden units during optimization

Efficient identification of the optimal number of hidden units

Higher test likelihoods compared to random initialization

Abstract

In this work, we propose an infinite restricted Boltzmann machine~(RBM), whose maximum likelihood estimation~(MLE) corresponds to a constrained convex optimization. We consider the Frank-Wolfe algorithm to solve the program, which provides a sparse solution that can be interpreted as inserting a hidden unit at each iteration, so that the optimization process takes the form of a sequence of finite models of increasing complexity. As a side benefit, this can be used to easily and efficiently identify an appropriate number of hidden units during the optimization. The resulting model can also be used as an initialization for typical state-of-the-art RBM training algorithms such as contrastive divergence, leading to models with consistently higher test likelihood than random initialization.