Sparse Boltzmann Machines with Structure Learning as Applied to Text Analysis

TL;DR

This paper introduces Sparse Boltzmann Machines with structure learning for text analysis, demonstrating improved interpretability and model fit over traditional RBMs by learning sparse connections between hidden and visible units.

Contribution

It presents a novel structure learning method for Sparse Boltzmann Machines applied to text, enhancing interpretability and model performance compared to standard RBMs.

Findings

Improved model fit over traditional RBMs

Enhanced interpretability of the learned models

Effective application to unsupervised text analysis

Abstract

We are interested in exploring the possibility and benefits of structure learning for deep models. As the first step, this paper investigates the matter for Restricted Boltzmann Machines (RBMs). We conduct the study with Replicated Softmax, a variant of RBMs for unsupervised text analysis. We present a method for learning what we call Sparse Boltzmann Machines, where each hidden unit is connected to a subset of the visible units instead of all of them. Empirical results show that the method yields models with significantly improved model fit and interpretability as compared with RBMs where each hidden unit is connected to all visible units.