Learning Feature Hierarchies with Centered Deep Boltzmann Machines

TL;DR

This paper introduces a centered deep Boltzmann machine with a modified learning algorithm that improves training stability and enables the model to learn hierarchical data representations more effectively.

Contribution

The paper proposes a new centering technique for Deep Boltzmann Machines that enhances training and allows for joint layer learning without greedy pretraining.

Findings

Centered Deep Boltzmann Machines learn hierarchical data representations.

The new algorithm results in better conditioned Hessian during training.

The model demonstrates improved generative capabilities on real data.

Abstract

Deep Boltzmann machines are in principle powerful models for extracting the hierarchical structure of data. Unfortunately, attempts to train layers jointly (without greedy layer-wise pretraining) have been largely unsuccessful. We propose a modification of the learning algorithm that initially recenters the output of the activation functions to zero. This modification leads to a better conditioned Hessian and thus makes learning easier. We test the algorithm on real data and demonstrate that our suggestion, the centered deep Boltzmann machine, learns a hierarchy of increasingly abstract representations and a better generative model of data.