Concentration Inequalities for Stochastic Optimization of Unbounded Objective Functions with Application to Denoising Score Matching

TL;DR

This paper develops new concentration inequalities for unbounded stochastic optimization problems and applies them to derive error bounds for denoising score matching, highlighting benefits of sample-reuse techniques.

Contribution

It introduces novel concentration inequalities based on sample-dependent bounds and applies them to unbounded objectives in denoising score matching, a previously challenging setting.

Findings

New McDiarmid's inequality for unbounded functions

Rademacher complexity bounds for unbounded distributions

Quantified benefits of sample-reuse in algorithms like DSM

Abstract

We derive novel concentration inequalities that bound the statistical error for a large class of stochastic optimization problems, focusing on the case of unbounded objective functions. Our derivations utilize the following key tools: 1) A new form of McDiarmid's inequality that is based on sample-dependent one-component mean-difference bounds and which leads to a novel uniform law of large numbers result for unbounded functions. 2) A new Rademacher complexity bound for families of functions that satisfy an appropriate sample-dependent Lipschitz property, which allows for application to a large class of distributions with unbounded support. As an application of these results, we derive statistical error bounds for denoising score matching (DSM), an application that inherently requires one to consider unbounded objective functions and distributions with unbounded support, even in cases where the data distribution has bounded support. In addition, our results quantify the benefit of sample-reuse in algorithms that employ easily-sampled auxiliary random variables in addition to the training data, e.g., as in DSM, which uses auxiliary Gaussian random variables.