Consistent Learning by Composite Proximal Thresholding

TL;DR

This paper introduces a new regularization model for machine learning that allows incorporating various priors, proves the consistency of the estimators, and develops an efficient proximal thresholding algorithm with proven convergence properties.

Contribution

It proposes a flexible composite regularization framework and an error-tolerant proximal algorithm with convergence guarantees for infinite-dimensional learning problems.

Findings

Estimators are statistically consistent.

The proposed algorithm converges with a rate of o(1/m).

New asymptotic results for proximal forward-backward splitting.

Abstract

We investigate the modeling and the numerical solution of machine learning problems with prediction functions which are linear combinations of elements of a possibly infinite-dimensional dictionary. We propose a novel flexible composite regularization model, which makes it possible to incorporate various priors on the coefficients of the prediction function, including sparsity and hard constraints. We show that the estimators obtained by minimizing the regularized empirical risk are consistent in a statistical sense, and we design an error-tolerant composite proximal thresholding algorithm for computing such estimators. New results on the asymptotic behavior of the proximal forward-backward splitting method are derived and exploited to establish the convergence properties of the proposed algorithm. In particular, our method features a $o(1/m)$ convergence rate in objective values.