Proximal Operators of Sorted Nonconvex Penalties

TL;DR

This paper explores efficient computation of proximal operators for sorted nonconvex penalties, enabling improved sparse signal recovery and variable grouping in generalized linear models.

Contribution

It introduces methods to compute proximal operators for a family of sorted nonconvex penalties, extending the applicability of proximal algorithms to these regularizers.

Findings

Proximal operators for sorted MCP and SCAD can be computed exactly using the PAV algorithm.

A modified PAV algorithm efficiently computes proximal operators for sorted Lq penalties with q in (0,1).

Experimental results show practical benefits of using sorted nonconvex penalties in signal recovery.

Abstract

This work studies the problem of sparse signal recovery with automatic grouping of variables. To this end, we investigate sorted nonsmooth penalties as a regularization approach for generalized linear models. We focus on a family of sorted nonconvex penalties which generalizes the Sorted L1 Norm (SLOPE). These penalties are designed to promote clustering of variables due to their sorted nature, while the nonconvexity reduces the shrinkage of coefficients. Our goal is to provide efficient ways to compute their proximal operator, enabling the use of popular proximal algorithms to solve composite optimization problems with this choice of sorted penalties. We distinguish between two classes of problems: the weakly convex case where computing the proximal operator remains a convex problem, and the nonconvex case where computing the proximal operator becomes a challenging nonconvex combinatorial problem. For the weakly convex case (e.g. sorted MCP and SCAD), we explain how the Pool Adjacent Violators (PAV) algorithm can exactly compute the proximal operator. For the nonconvex case (e.g. sorted Lq with q in ]0,1[), we show that a slight modification of this algorithm turns out to be remarkably efficient to tackle the computation of the proximal operator. We also present new theoretical insights on the minimizers of the nonconvex proximal problem. We demonstrate the practical interest of using such penalties on several experiments.