Towards universal neural nets: Gibbs machines and ACE

TL;DR

This paper introduces Gibbs machines and ACE, a class of universal generative neural nets rooted in physics principles, capable of incremental learning and including symmetries, with state-of-the-art results on MNIST.

Contribution

It presents a physics-inspired framework for generative neural nets that can incrementally learn and incorporate symmetries, unifying variational auto-encoders and other models.

Findings

Gibbs machines connect neural nets with statistical physics and information geometry.

ACE achieves state-of-the-art performance in classification and density estimation on MNIST.

The framework supports incremental learning of features and symmetries.

Abstract

We study from a physics viewpoint a class of generative neural nets, Gibbs machines, designed for gradual learning. While including variational auto-encoders, they offer a broader universal platform for incrementally adding newly learned features, including physical symmetries. Their direct connection to statistical physics and information geometry is established. A variational Pythagorean theorem justifies invoking the exponential/Gibbs class of probabilities for creating brand new objects. Combining these nets with classifiers, gives rise to a brand of universal generative neural nets - stochastic auto-classifier-encoders (ACE). ACE have state-of-the-art performance in their class, both for classification and density estimation for the MNIST data set.