Score Change of Variables

TL;DR

This paper introduces a general change of variables formula for score functions, enabling new methods for diffusion models and high-dimensional density estimation through flexible transformations.

Contribution

It derives a novel score transformation formula and applies it to develop reverse-time diffusion techniques and generalized sliced score matching.

Findings

Reverse-time Itô lemma for score-based diffusion models

Decoupling forward and reverse processes in diffusion models

Enhanced high-dimensional density estimation with generalized sliced score matching

Abstract

We derive a general change of variables formula for score functions, showing that for a smooth, invertible transformation $\mathbf{y} = \phi(\mathbf{x})$, the transformed score function $\nabla_{\mathbf{y}} \log q(\mathbf{y})$ can be expressed directly in terms of $\nabla_{\mathbf{x}} \log p(\mathbf{x})$. Using this result, we develop two applications: First, we establish a reverse-time It\^o lemma for score-based diffusion models, allowing the use of $\nabla_{\mathbf{x}} \log p_t(\mathbf{x})$ to reverse an SDE in the transformed space without directly learning $\nabla_{\mathbf{y}} \log q_t(\mathbf{y})$. This approach enables training diffusion models in one space but sampling in another, effectively decoupling the forward and reverse processes. Second, we introduce generalized sliced score matching, extending traditional sliced score matching from linear projections to arbitrary smooth transformations. This provides greater flexibility in high-dimensional density estimation. We demonstrate these theoretical advances through applications to diffusion on the probability simplex and empirically compare our generalized score matching approach against traditional sliced score matching methods.