Sliced Score Matching: A Scalable Approach to Density and Score Estimation

TL;DR

Sliced score matching introduces a scalable method for density and score estimation that leverages Hessian-vector products, enabling the use of complex models and high-dimensional data efficiently.

Contribution

It proposes sliced score matching, a novel approach that simplifies Hessian computations, allowing for scalable density estimation with deep models and high-dimensional data.

Findings

Effective learning of deep energy-based models

Accurate score estimation for variational inference

Applicable to training Wasserstein Auto-Encoders

Abstract

Score matching is a popular method for estimating unnormalized statistical models. However, it has been so far limited to simple, shallow models or low-dimensional data, due to the difficulty of computing the Hessian of log-density functions. We show this difficulty can be mitigated by projecting the scores onto random vectors before comparing them. This objective, called sliced score matching, only involves Hessian-vector products, which can be easily implemented using reverse-mode automatic differentiation. Therefore, sliced score matching is amenable to more complex models and higher dimensional data compared to score matching. Theoretically, we prove the consistency and asymptotic normality of sliced score matching estimators. Moreover, we demonstrate that sliced score matching can be used to learn deep score estimators for implicit distributions. In our experiments, we show sliced score matching can learn deep energy-based models effectively, and can produce accurate score estimates for applications such as variational inference with implicit distributions and training Wasserstein Auto-Encoders.