Sparse Polyak with optimal thresholding operators for high-dimensional M-estimation

TL;DR

This paper introduces an improved Sparse Polyak algorithm with optimal thresholding for high-dimensional M-estimation, achieving better sparsity and accuracy without sacrificing scalability.

Contribution

The authors develop a variant of Sparse Polyak that maintains optimal scaling, enhances sparsity, and improves statistical accuracy in high-dimensional settings.

Findings

Retains performance as ambient dimension increases

Produces sparser solutions

Achieves more accurate estimations

Abstract

We propose and analyze a variant of Sparse Polyak for high dimensional M-estimation problems. Sparse Polyak proposes a novel adaptive step-size rule tailored to suitably estimate the problem's curvature in the high-dimensional setting, guaranteeing that the algorithm's performance does not deteriorate when the ambient dimension increases. However, convergence guarantees can only be obtained by sacrificing solution sparsity and statistical accuracy. In this work, we introduce a variant of Sparse Polyak that retains its desirable scaling properties with respect to the ambient dimension while obtaining sparser and more accurate solutions.