Smoothing fast iterative hard thresholding algorithm for $\ell_0$ regularized nonsmooth convex regression problem

TL;DR

This paper introduces a novel smoothing fast iterative hard thresholding algorithm for solving nonsmooth convex sparse regression problems with cardinality constraints, providing convergence guarantees and rate analysis.

Contribution

It proposes a new SFIHT algorithm combining smoothing, extrapolation, and hard thresholding, with convergence analysis and rates for nonsmooth convex regression.

Findings

Convergence to local minimizers under certain conditions

Achieves an $O(rac{ ext{ln} k}{k})$ convergence rate for the loss

Attains an $o(k^{-2})$ rate when the loss is smooth

Abstract

We investigate a class of constrained sparse regression problem with cardinality penalty, where the feasible set is defined by box constraint, and the loss function is convex, but not necessarily smooth. First, we put forward a smoothing fast iterative hard thresholding (SFIHT) algorithm for solving such optimization problems, which combines smoothing approximations, extrapolation techniques and iterative hard thresholding methods. The extrapolation coefficients can be chosen to satisfy $\sup_k \beta_k=1$ in the proposed algorithm. We discuss the convergence behavior of the algorithm with different extrapolation coefficients, and give sufficient conditions to ensure that any accumulation point of the iterates is a local minimizer of the original cardinality penalized problem. In particular, for a class of fixed extrapolation coefficients, we discuss several different update rules of the smoothing parameter and obtain the convergence rate of $O(\ln k/k)$ on the loss and objective function values. Second, we consider the case in which the loss function is Lipschitz continuously differentiable, and develop a fast iterative hard thresholding (FIHT) algorithm to solve it. We prove that the iterates of FIHT converge to a local minimizer of the problem that satisfies a desirable lower bound property. Moreover, we show that the convergence rate of loss and objective function values are $o(k^{-2})$. Finally, some numerical examples are presented to illustrate the theoretical results.