Learning normalized image densities via dual score matching

TL;DR

This paper introduces a novel framework for learning normalized energy models using dual score matching, inspired by diffusion models, achieving competitive likelihoods and strong generalization on ImageNet64.

Contribution

It proposes a new dual score matching approach for normalized energy models, enabling consistent likelihood estimation across noise levels and demonstrating strong generalization capabilities.

Findings

Achieved state-of-the-art comparable negative log likelihood on ImageNet64.

Log probabilities from different trained networks are nearly identical, showing strong generalization.

Image content significantly affects local neighborhood dimensionality and probability, challenging traditional assumptions.

Abstract

Learning probability models from data is at the heart of many machine learning endeavors, but is notoriously difficult due to the curse of dimensionality. We introduce a new framework for learning \emph{normalized} energy (log probability) models that is inspired by diffusion generative models, which rely on networks optimized to estimate the score. We modify a score network architecture to compute an energy while preserving its inductive biases. The gradient of this energy network with respect to its input image is the score of the learned density, which can be optimized using a denoising objective. Importantly, the gradient with respect to the noise level provides an additional score that can be optimized with a novel secondary objective, ensuring consistent and normalized energies across noise levels. We train an energy network with this \emph{dual} score matching objective on the ImageNet64 dataset, and obtain a cross-entropy (negative log likelihood) value comparable to the state of the art. We further validate our approach by showing that our energy model \emph{strongly generalizes}: log probabilities estimated with two networks trained on non-overlapping data subsets are nearly identical. Finally, we demonstrate that both image probability and dimensionality of local neighborhoods vary substantially depending on image content, in contrast with conventional assumptions such as concentration of measure or support on a low-dimensional manifold.