Generalising the Discriminative Restricted Boltzmann Machine

TL;DR

This paper extends the Discriminative Restricted Boltzmann Machine (DRBM) to incorporate various hidden unit distributions beyond the original Bernoulli, demonstrating improved performance on multiple classification benchmarks.

Contribution

It provides a theoretical generalisation of the DRBM allowing different hidden unit distributions, including Binomial and {-1, +1}-Bernoulli, with empirical validation.

Findings

Each DRBM variant outperforms others on at least one dataset.

The generalisation improves flexibility in model design.

Results vary depending on the dataset and distribution used.

Abstract

We present a novel theoretical result that generalises the Discriminative Restricted Boltzmann Machine (DRBM). While originally the DRBM was defined assuming the {0, 1}-Bernoulli distribution in each of its hidden units, this result makes it possible to derive cost functions for variants of the DRBM that utilise other distributions, including some that are often encountered in the literature. This is illustrated with the Binomial and {-1, +1}-Bernoulli distributions here. We evaluate these two DRBM variants and compare them with the original one on three benchmark datasets, namely the MNIST and USPS digit classification datasets, and the 20 Newsgroups document classification dataset. Results show that each of the three compared models outperforms the remaining two in one of the three datasets, thus indicating that the proposed theoretical generalisation of the DRBM may be valuable in practice.