Denoising Deep Generative Models

TL;DR

This paper investigates the limitations of likelihood-based deep generative models on low-dimensional data and proposes two noise-based denoising methods, but finds they often underperform compared to alternative solutions.

Contribution

It introduces two novel noise-based denoising techniques to address the dimensionality mismatch in deep generative models, highlighting their limited empirical effectiveness.

Findings

Noise addition sometimes improves model performance

Other methods are more empirically effective

Denoising mechanisms can sample as if no noise was added

Abstract

Likelihood-based deep generative models have recently been shown to exhibit pathological behaviour under the manifold hypothesis as a consequence of using high-dimensional densities to model data with low-dimensional structure. In this paper we propose two methodologies aimed at addressing this problem. Both are based on adding Gaussian noise to the data to remove the dimensionality mismatch during training, and both provide a denoising mechanism whose goal is to sample from the model as though no noise had been added to the data. Our first approach is based on Tweedie's formula, and the second on models which take the variance of added noise as a conditional input. We show that surprisingly, while well motivated, these approaches only sporadically improve performance over not adding noise, and that other methods of addressing the dimensionality mismatch are more empirically adequate.