Rademacher Complexity of the Restricted Boltzmann Machine

TL;DR

This paper analyzes the Rademacher complexity of restricted Boltzmann machines, revealing that practical training procedures like CD-1 increase their complexity, which impacts their theoretical understanding.

Contribution

It provides the first analysis of Rademacher complexity for both asymptotic and practical RBMs, highlighting the effect of CD-1 training.

Findings

Practical CD-1 training increases Rademacher complexity.

Theoretical analysis of asymptotic RBMs.

Suggests future work on VC dimension of CD-1 functions.

Abstract

Boltzmann machine, as a fundamental construction block of deep belief network and deep Boltzmann machines, is widely used in deep learning community and great success has been achieved. However, theoretical understanding of many aspects of it is still far from clear. In this paper, we studied the Rademacher complexity of both the asymptotic restricted Boltzmann machine and the practical implementation with single-step contrastive divergence (CD-1) procedure. Our results disclose the fact that practical implementation training procedure indeed increased the Rademacher complexity of restricted Boltzmann machines. A further research direction might be the investigation of the VC dimension of a compositional function used in the CD-1 procedure.