The Gaussian-Multinoulli Restricted Boltzmann Machine: A Potts Model Extension of the GRBM

TL;DR

This paper introduces the Gaussian-Multinoulli RBM, extending the Gaussian-Bernoulli RBM with categorical units to better model discrete, structured representations, and demonstrates its effectiveness on memory and analogy tasks.

Contribution

The paper presents the GM-RBM, a novel energy-based model with categorical hidden units, offering a richer latent space and improved performance over binary models.

Findings

GM-RBM achieves competitive recall on memory benchmarks.

Discrete categorical units outperform binary units at equal capacity.

Efficient implementation of q-ary latent variables is demonstrated.

Abstract

Many real-world tasks, from associative memory to symbolic reasoning, benefit from discrete, structured representations that standard continuous latent models can struggle to express. We introduce the Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM), a generative energy-based model that extends the Gaussian-Bernoulli RBM (GB-RBM) by replacing binary hidden units with q-state categorical (Potts) units, yielding a richer latent state space for multivalued concepts. We provide a self-contained derivation of the energy, conditional distributions, and learning rules, and detail practical training choices (contrastive divergence with temperature annealing and intra-slot diversity constraints) that avoid state collapse. To separate architectural effects from sheer latent capacity, we evaluate under both capacity-matched and parameter-matched setups, comparing GM-RBM with GB-RBM configured to have the same number of possible latent assignments. On analogical recall and structured memory benchmarks, GM-RBM achieves competitive, and in several regimes improved, recall at equal capacity with comparable training cost, despite using only Gibbs updates. The discrete q-ary formulation is also amenable to efficient implementation. These results clarify when categorical hidden units provide a simple, scalable alternative to binary latents for discrete inference within tractable RBMs.