First-Order Sparse Convex Optimization: Better Rates with Sparse Updates

TL;DR

This paper introduces sparse update methods for convex optimization with sparse solutions, achieving faster convergence rates and reduced per-iteration complexity, making high-dimensional problems more tractable.

Contribution

It demonstrates that linear convergence with improved condition number dependence can be achieved using only sparse updates, simplifying implementation and enhancing efficiency.

Findings

Achieves linear convergence with sparse updates

Reduces per-iteration computational complexity

Easier to implement than previous methods

Abstract

It was recently established that for convex optimization problems with sparse optimal solutions (be it entry-wise sparsity or matrix rank-wise sparsity) it is possible to design first-order methods with linear convergence rates that depend on an improved mixed-norm condition number of the form $\frac{\beta_1{}s}{\alpha_2}$, where $\beta_1$ is the $\ell_1$-Lipschitz continuity constant of the gradient, $\alpha_2$ is the $\ell_2$-quadratic growth constant, and $s$ is the sparsity of optimal solutions. However, beyond the improved convergence rate, these methods are unable to leverage the sparsity of optimal solutions towards improving the runtime of each iteration as well, which may still be prohibitively high for high-dimensional problems. In this work, we establish that linear convergence rates which depend on this improved condition number can be obtained using only sparse updates, which may result in overall significantly improved running times. Moreover, our methods are considerably easier to implement.