Multimeasurement Generative Models

TL;DR

This paper introduces multimeasurement generative models that map the problem of sampling from an unknown distribution to learning a smoother, higher-dimensional density, enabling efficient sampling and establishing a novel connection with denoising autoencoders.

Contribution

It formulates a new framework using M-densities and multimeasurement noise models, deriving closed-form Bayes estimators and linking denoising autoencoders with empirical Bayes theory.

Findings

Effective sampling from complex distributions using walk-jump sampling.

Demonstrated stability and fast mixing of Markov chains in high-dimensional image datasets.

Established a theoretical connection between denoising autoencoders and empirical Bayes methods.

Abstract

We formally map the problem of sampling from an unknown distribution with a density in $\mathbb{R}^d$ to the problem of learning and sampling a smoother density in $\mathbb{R}^{Md}$ obtained by convolution with a fixed factorial kernel: the new density is referred to as M-density and the kernel as multimeasurement noise model (MNM). The M-density in $\mathbb{R}^{Md}$ is smoother than the original density in $\mathbb{R}^d$, easier to learn and sample from, yet for large $M$ the two problems are mathematically equivalent since clean data can be estimated exactly given a multimeasurement noisy observation using the Bayes estimator. To formulate the problem, we derive the Bayes estimator for Poisson and Gaussian MNMs in closed form in terms of the unnormalized M-density. This leads to a simple least-squares objective for learning parametric energy and score functions. We present various parametrization schemes of interest including one in which studying Gaussian M-densities directly leads to multidenoising autoencoders--this is the first theoretical connection made between denoising autoencoders and empirical Bayes in the literature. Samples in $\mathbb{R}^d$ are obtained by walk-jump sampling (Saremi & Hyvarinen, 2019) via underdamped Langevin MCMC (walk) to sample from M-density and the multimeasurement Bayes estimation (jump). We study permutation invariant Gaussian M-densities on MNIST, CIFAR-10, and FFHQ-256 datasets, and demonstrate the effectiveness of this framework for realizing fast-mixing stable Markov chains in high dimensions.