Precise asymptotics of reweighted least-squares algorithms for linear diagonal networks

TL;DR

This paper provides a detailed asymptotic analysis of reweighted least-squares algorithms, including IRLS and neural network-inspired methods, demonstrating their efficiency and effectiveness in high-dimensional linear models with structured sparsity.

Contribution

It offers a unified asymptotic analysis for IRLS, lin-RFM, and linear diagonal neural networks, revealing their performance in high-dimensional settings with Gaussian covariates.

Findings

Algorithms achieve favorable performance in few iterations.

Reweighting schemes improve test error with structured sparsity.

Analysis applies to neural network-inspired and group-sparse recovery methods.

Abstract

The classical iteratively reweighted least-squares (IRLS) algorithm aims to recover an unknown signal from linear measurements by performing a sequence of weighted least squares problems, where the weights are recursively updated at each step. Varieties of this algorithm have been shown to achieve favorable empirical performance and theoretical guarantees for sparse recovery and $\ell_p$-norm minimization. Recently, some preliminary connections have also been made between IRLS and certain types of non-convex linear neural network architectures that are observed to exploit low-dimensional structure in high-dimensional linear models. In this work, we provide a unified asymptotic analysis for a family of algorithms that encompasses IRLS, the recently proposed lin-RFM algorithm (which was motivated by feature learning in neural networks), and the alternating minimization algorithm on linear diagonal neural networks. Our analysis operates in a "batched" setting with i.i.d. Gaussian covariates and shows that, with appropriately chosen reweighting policy, the algorithm can achieve favorable performance in only a handful of iterations. We also extend our results to the case of group-sparse recovery and show that leveraging this structure in the reweighting scheme provably improves test error compared to coordinate-wise reweighting.