Natural Reweighted Wake-Sleep

TL;DR

This paper introduces Natural Reweighted Wake-Sleep (NRWS) and Natural Bidirectional Helmholtz Machine (NBiHM), leveraging the Fisher information matrix's structure for efficient natural gradient training of Helmholtz Machines, leading to improved convergence and likelihood.

Contribution

The paper proposes a novel natural gradient-based training algorithm for Helmholtz Machines that exploits the Fisher matrix's structure for efficiency and improved performance.

Findings

NRWS and NBiHM outperform non-geometric baselines.

Faster convergence and higher log-likelihood values achieved.

Efficient natural gradient computation without approximations.

Abstract

Helmholtz Machines (HMs) are a class of generative models composed of two Sigmoid Belief Networks (SBNs), acting respectively as an encoder and a decoder. These models are commonly trained using a two-step optimization algorithm called Wake-Sleep (WS) and more recently by improved versions, such as Reweighted Wake-Sleep (RWS) and Bidirectional Helmholtz Machines (BiHM). The locality of the connections in an SBN induces sparsity in the Fisher Information Matrices associated to the probabilistic models, in the form of a finely-grained block-diagonal structure. In this paper we exploit this property to efficiently train SBNs and HMs using the natural gradient. We present a novel algorithm, called Natural Reweighted Wake-Sleep (NRWS), that corresponds to the geometric adaptation of its standard version. In a similar manner, we also introduce Natural Bidirectional Helmholtz Machine (NBiHM). Differently from previous work, we will show how for HMs the natural gradient can be efficiently computed without the need of introducing any approximation in the structure of the Fisher information matrix. The experiments performed on standard datasets from the literature show a consistent improvement of NRWS and NBiHM not only with respect to their non-geometric baselines but also with respect to state-of-the-art training algorithms for HMs. The improvement is quantified both in terms of speed of convergence as well as value of the log-likelihood reached after training.