On better training the infinite restricted Boltzmann machines

TL;DR

This paper introduces a novel training strategy for infinite restricted Boltzmann machines (iRBMs) that involves randomly regrouping hidden units to accelerate convergence and improve generalization, making iRBMs more practical.

Contribution

The paper proposes a new training method for iRBMs that reduces training time and enhances generalization by random permutation of hidden units during learning.

Findings

Training speed is significantly improved.

Model generalization is enhanced.

Effective on datasets like MNIST and CalTech101.

Abstract

The infinite restricted Boltzmann machine (iRBM) is an extension of the classic RBM. It enjoys a good property of automatically deciding the size of the hidden layer according to specific training data. With sufficient training, the iRBM can achieve a competitive performance with that of the classic RBM. However, the convergence of learning the iRBM is slow, due to the fact that the iRBM is sensitive to the ordering of its hidden units, the learned filters change slowly from the left-most hidden unit to right. To break this dependency between neighboring hidden units and speed up the convergence of training, a novel training strategy is proposed. The key idea of the proposed training strategy is randomly regrouping the hidden units before each gradient descent step. Potentially, a mixing of infinite many iRBMs with different permutations of the hidden units can be achieved by this learning method, which has a similar effect of preventing the model from over-fitting as the dropout. The original iRBM is also modified to be capable of carrying out discriminative training. To evaluate the impact of our method on convergence speed of learning and the model's generalization ability, several experiments have been performed on the binarized MNIST and CalTech101 Silhouettes datasets. Experimental results indicate that the proposed training strategy can greatly accelerate learning and enhance generalization ability of iRBMs.