Boltzmann Machine Learning with the Latent Maximum Entropy Principle

TL;DR

This paper introduces the latent maximum entropy principle for Boltzmann machine learning, offering a new inference method that outperforms traditional maximum likelihood estimation, especially with limited data.

Contribution

It proposes the latent maximum entropy principle and develops new algorithms, including a robust, fast EM variant for Boltzmann machine parameter estimation.

Findings

LME-based estimation outperforms MLE in small data scenarios

New algorithms improve robustness and speed of learning

LME provides a different inference paradigm from existing principles

Abstract

We present a new statistical learning paradigm for Boltzmann machines based on a new inference principle we have proposed: the latent maximum entropy principle (LME). LME is different both from Jaynes maximum entropy principle and from standard maximum likelihood estimation.We demonstrate the LME principle BY deriving new algorithms for Boltzmann machine parameter estimation, and show how robust and fast new variant of the EM algorithm can be developed.Our experiments show that estimation based on LME generally yields better results than maximum likelihood estimation, particularly when inferring hidden units from small amounts of data.