When Does a Mixture of Products Contain a Product of Mixtures?

TL;DR

This paper explores the theoretical properties of RBMs and their relation to discrete mathematics and convex geometry, providing insights into their representational power and the complexity of mixtures versus products.

Contribution

It establishes formal relations between RBM-representable distributions and mathematical structures, and justifies the intuition that products of mixtures are more efficient than mixtures of products.

Findings

Mixtures of products require exponentially more parameters than products of mixtures.

Relations between RBMs and geometric structures like zonotopes are established.

Results justify the efficiency of distributed representations in RBMs.

Abstract

We derive relations between theoretical properties of restricted Boltzmann machines (RBMs), popular machine learning models which form the building blocks of deep learning models, and several natural notions from discrete mathematics and convex geometry. We give implications and equivalences relating RBM-representable probability distributions, perfectly reconstructible inputs, Hamming modes, zonotopes and zonosets, point configurations in hyperplane arrangements, linear threshold codes, and multi-covering numbers of hypercubes. As a motivating application, we prove results on the relative representational power of mixtures of product distributions and products of mixtures of pairs of product distributions (RBMs) that formally justify widely held intuitions about distributed representations. In particular, we show that a mixture of products requiring an exponentially larger number of parameters is needed to represent the probability distributions which can be obtained as products of mixtures.