Small noise analysis for Tikhonov and RKHS regularizations

TL;DR

This paper develops a small noise analysis framework for Tikhonov and RKHS regularizations, revealing stability issues and proposing adaptive fractional RKHS regularizers that improve convergence rates in ill-posed inverse problems.

Contribution

It introduces a novel small noise analysis framework and proposes adaptive fractional RKHS regularizers to enhance stability and convergence in ill-posed problems.

Findings

Over-smoothing with fractional RKHSs yields optimal convergence rates.

Conventional L2-regularizer may be unstable under small noise.

Adaptive fractional regularizers can cover Tikhonov and RKHS methods.

Abstract

Regularization plays a pivotal role in ill-posed machine learning and inverse problems. However, the fundamental comparative analysis of various regularization norms remains open. We establish a small noise analysis framework to assess the effects of norms in Tikhonov and RKHS regularizations, in the context of ill-posed linear inverse problems with Gaussian noise. This framework studies the convergence rates of regularized estimators in the small noise limit and reveals the potential instability of the conventional L2-regularizer. We solve such instability by proposing an innovative class of adaptive fractional RKHS regularizers, which covers the L2 Tikhonov and RKHS regularizations by adjusting the fractional smoothness parameter. A surprising insight is that over-smoothing via these fractional RKHSs consistently yields optimal convergence rates, but the optimal hyper-parameter may decay too fast to be selected in practice.