Denoising Score Matching with Random Fourier Features

TL;DR

This paper introduces a novel denoising score matching method using Random Fourier Features and Kernel Exponential Families, enabling efficient density estimation and scalability to high-dimensional data.

Contribution

It derives an analytical expression for denoising score matching with kernel exponential families using Random Fourier Features, improving computational efficiency and tuning simplicity.

Findings

Comparable quality to existing methods in benchmarks

Faster computation enabling high-dimensional data scaling

Explicit dependence on noise variance for easier tuning

Abstract

The density estimation is one of the core problems in statistics. Despite this, existing techniques like maximum likelihood estimation are computationally inefficient due to the intractability of the normalizing constant. For this reason an interest to score matching has increased being independent on the normalizing constant. However, such estimator is consistent only for distributions with the full space support. One of the approaches to make it consistent is to add noise to the input data which is called Denoising Score Matching. In this work we derive analytical expression for the Denoising Score matching using the Kernel Exponential Family as a model distribution. The usage of the kernel exponential family is motivated by the richness of this class of densities. To tackle the computational complexity we use Random Fourier Features based approximation of the kernel function. The analytical expression allows to drop additional regularization terms based on the higher-order derivatives as they are already implicitly included. Moreover, the obtained expression explicitly depends on the noise variance, so the validation loss can be straightforwardly used to tune the noise level. Along with benchmark experiments, the model was tested on various synthetic distributions to study the behaviour of the model in different cases. The empirical study shows comparable quality to the competing approaches, while the proposed method being computationally faster. The latter one enables scaling up to complex high-dimensional data.