Stability and Risk Bounds of Iterative Hard Thresholding

TL;DR

This paper develops a new theoretical framework for understanding the generalization performance of the Iterative Hard Thresholding algorithm in sparse recovery, providing convergence rates and stability bounds with practical implications.

Contribution

It introduces a novel sparse generalization theory for IHT based on algorithmic stability, deriving new convergence rates and applying them to linear and logistic regression models.

Findings

IHT achieves an $ ilde{O}(n^{-1/2})$ convergence rate in sparse excess risk.

A tighter $ ilde{O}(n^{-1/2})$ bound is possible with iteration stability.

Fast rate $ ilde{O}(n^{-1}k( ext{log}^3 n + ext{log} p))$ under strong convexity and signal conditions.

Abstract

In this paper, we analyze the generalization performance of the Iterative Hard Thresholding (IHT) algorithm widely used for sparse recovery problems. The parameter estimation and sparsity recovery consistency of IHT has long been known in compressed sensing. From the perspective of statistical learning, another fundamental question is how well the IHT estimation would predict on unseen data. This paper makes progress towards answering this open question by introducing a novel sparse generalization theory for IHT under the notion of algorithmic stability. Our theory reveals that: 1) under natural conditions on the empirical risk function over $n$ samples of dimension $p$, IHT with sparsity level $k$ enjoys an $\mathcal{\tilde O}(n^{-1/2}\sqrt{k\log(n)\log(p)})$ rate of convergence in sparse excess risk; 2) a tighter $\mathcal{\tilde O}(n^{-1/2}\sqrt{\log(n)})$ bound can be established by imposing an additional iteration stability condition on a hypothetical IHT procedure invoked to the population risk; and 3) a fast rate of order $\mathcal{\tilde O}\left(n^{-1}k(\log^3(n)+\log(p))\right)$ can be derived for strongly convex risk function under proper strong-signal conditions. The results have been substantialized to sparse linear regression and sparse logistic regression models to demonstrate the applicability of our theory. Preliminary numerical evidence is provided to confirm our theoretical predictions.