Proximal Algorithms in Statistics and Machine Learning

TL;DR

This paper introduces novel proximal algorithms for statistical learning that leverage envelope representations and closed-form solutions to efficiently optimize complex, non-smooth, and non-convex objectives in regression models.

Contribution

It develops new proximal algorithms based on envelope representations for optimizing composite functions in statistics and machine learning, including non-convex penalties.

Findings

Effective algorithms for regularized logistic and Poisson regression.

Proposed methods handle non-convex and non-smooth objectives.

Discussion on convergence and acceleration of non-descent algorithms.

Abstract

In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Forward-Backward, Douglas-Rachford and Half-Quadratic envelopes. Envelope representations lead to novel proximal algorithms for statistical optimisation of composite objective functions which include both non-smooth and non-convex objectives. We illustrate our methodology with regularized Logistic and Poisson regression and non-convex bridge penalties with a fused lasso norm. We provide a discussion of convergence of non-descent algorithms with acceleration and for non-convex functions. Finally, we provide directions for future research.