A General Framework for Fast Stagewise Algorithms
Ryan J. Tibshirani

TL;DR
This paper introduces a versatile framework for fast stagewise algorithms that extend the simple, iterative approach of forward stagewise regression to a variety of regularization problems beyond sparsity, demonstrating broad applicability.
Contribution
It develops a general stagewise estimation framework applicable to diverse regularization tasks, enabling efficient algorithms for complex models like group learning and matrix completion.
Findings
Framework unifies various regularization problems under stagewise methods.
Algorithms are computationally efficient and applicable to high-dimensional data.
Broad applicability demonstrated across multiple machine learning tasks.
Abstract
Forward stagewise regression follows a very simple strategy for constructing a sequence of sparse regression estimates: it starts with all coefficients equal to zero, and iteratively updates the coefficient (by a small amount ) of the variable that achieves the maximal absolute inner product with the current residual. This procedure has an interesting connection to the lasso: under some conditions, it is known that the sequence of forward stagewise estimates exactly coincides with the lasso path, as the step size goes to zero. Furthermore, essentially the same equivalence holds outside of least squares regression, with the minimization of a differentiable convex loss function subject to an norm constraint (the stagewise algorithm now updates the coefficient corresponding to the maximal absolute component of the gradient). Even when they do not match their…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSparse and Compressive Sensing Techniques · Numerical methods in inverse problems · Statistical Methods and Inference
