On the Suboptimality of Proximal Gradient Descent for $\ell^{0}$ Sparse Approximation

TL;DR

This paper analyzes the limitations of proximal gradient descent for 0 sparse approximation, introduces randomized acceleration algorithms, and proves their effectiveness in reducing computation while maintaining bounded suboptimality.

Contribution

It provides theoretical bounds for PGD's suboptimality under weaker conditions and proposes randomized algorithms that accelerate optimization with provable guarantees.

Findings

PGD's suboptimality gap is bounded under weaker conditions than RIP.

Randomized algorithms significantly reduce computational cost.

Suboptimal solutions from randomized methods still have provable bounds.

Abstract

We study the proximal gradient descent (PGD) method for $\ell^{0}$ sparse approximation problem as well as its accelerated optimization with randomized algorithms in this paper. We first offer theoretical analysis of PGD showing the bounded gap between the sub-optimal solution by PGD and the globally optimal solution for the $\ell^{0}$ sparse approximation problem under conditions weaker than Restricted Isometry Property widely used in compressive sensing literature. Moreover, we propose randomized algorithms to accelerate the optimization by PGD using randomized low rank matrix approximation (PGD-RMA) and randomized dimension reduction (PGD-RDR). Our randomized algorithms substantially reduces the computation cost of the original PGD for the $\ell^{0}$ sparse approximation problem, and the resultant sub-optimal solution still enjoys provable suboptimality, namely, the sub-optimal solution to the reduced problem still has bounded gap to the globally optimal solution to the original problem.