Soft-Deep Boltzmann Machines

TL;DR

This paper critically examines the representational power of deep Boltzmann machines (DBMs), revealing their limitations, and introduces soft-deep BMs (sDBMs) as a more efficient alternative that outperforms existing models in generative tasks.

Contribution

The paper provides a theoretical and empirical analysis of DBMs' limitations and proposes the novel sDBM architecture that better exploits distributed representations.

Findings

DBMs have limited representational efficiency despite their depth.

sDBMs outperform DBMs and other models in generative tasks.

A new measure for the efficiency of distributed representations is introduced.

Abstract

We present a layered Boltzmann machine (BM) that can better exploit the advantages of a distributed representation. It is widely believed that deep BMs (DBMs) have far greater representational power than its shallow counterpart, restricted Boltzmann machines (RBMs). However, this expectation on the supremacy of DBMs over RBMs has not ever been validated in a theoretical fashion. In this paper, we provide both theoretical and empirical evidences that the representational power of DBMs can be actually rather limited in taking advantages of distributed representations. We propose an approximate measure for the representational power of a BM regarding to the efficiency of a distributed representation. With this measure, we show a surprising fact that DBMs can make inefficient use of distributed representations. Based on these observations, we propose an alternative BM architecture, which we dub soft-deep BMs (sDBMs). We show that sDBMs can more efficiently exploit the distributed representations in terms of the measure. Experiments demonstrate that sDBMs outperform several state-of-the-art models, including DBMs, in generative tasks on binarized MNIST and Caltech-101 silhouettes.