Regularization and the small-ball method I: sparse recovery

TL;DR

This paper develops a unified framework for analyzing regularization methods in learning, providing new bounds on estimation errors and insights into sparsity and subdifferential properties, extending results for LASSO, SLOPE, and trace norm.

Contribution

It introduces a general approach to bound estimation errors in convex regularization, linking sparsity notions to subdifferential sizes, and extends existing estimates to new regularizers.

Findings

Unified bounds for regularization procedures

Insights into sparsity and subdifferential relationships

Extended estimates for LASSO, SLOPE, and trace norm

Abstract

We obtain bounds on estimation error rates for regularization procedures of the form \begin{equation*} \hat f \in {\rm argmin}_{f\in F}\left(\frac{1}{N}\sum_{i=1}^N\left(Y_i-f(X_i)\right)^2+\lambda \Psi(f)\right) \end{equation*} when $\Psi$ is a norm and $F$ is convex. Our approach gives a common framework that may be used in the analysis of learning problems and regularization problems alike. In particular, it sheds some light on the role various notions of sparsity have in regularization and on their connection with the size of subdifferentials of $\Psi$ in a neighbourhood of the true minimizer. As `proof of concept' we extend the known estimates for the LASSO, SLOPE and trace norm regularization.