Optimal score estimation via empirical Bayes smoothing

TL;DR

This paper establishes the optimal rate for estimating the score function of a distribution from samples, demonstrating the curse of dimensionality and proposing a regularized estimator that achieves this rate.

Contribution

It introduces a regularized Gaussian kernel-based score estimator that attains the optimal convergence rate, supported by a new convergence analysis and minimax lower bounds.

Findings

Optimal rate of rac{2}{d+4} for score estimation

Regularized estimator achieves the optimal rate

Highlights the curse of dimensionality in score estimation

Abstract

We study the problem of estimating the score function of an unknown probability distribution $\rho^*$ from $n$ independent and identically distributed observations in $d$ dimensions. Assuming that $\rho^*$ is subgaussian and has a Lipschitz-continuous score function $s^*$, we establish the optimal rate of $\tilde \Theta(n^{-\frac{2}{d+4}})$ for this estimation problem under the loss function $\|\hat s - s^*\|^2_{L^2(\rho^*)}$ that is commonly used in the score matching literature, highlighting the curse of dimensionality where sample complexity for accurate score estimation grows exponentially with the dimension $d$. Leveraging key insights in empirical Bayes theory as well as a new convergence rate of smoothed empirical distribution in Hellinger distance, we show that a regularized score estimator based on a Gaussian kernel attains this rate, shown optimal by a matching minimax lower bound. We also discuss extensions to estimating $\beta$-H\"older continuous scores with $\beta \leq 1$, as well as the implication of our theory on the sample complexity of score-based generative models.