Step-Size Decay and Structural Stagnation in Greedy Sparse Learning

TL;DR

This paper investigates how overly rapid decay of step sizes in greedy algorithms can cause stagnation in sparse learning, providing theoretical bounds and empirical evidence to guide better step-size choices.

Contribution

It offers a theoretical analysis of step-size decay effects in greedy sparse learning, highlighting the impact of feature coherence on convergence.

Findings

Over-decaying step sizes induce structural stagnation.

Explicit lower bounds on residual norms are derived.

Numerical experiments confirm theoretical predictions.

Abstract

Greedy algorithms are central to sparse approximation and stage-wise learning methods such as matching pursuit and boosting. It is known that the Power-Relaxed Greedy Algorithm with step sizes $m^{-\alpha}$ may fail to converge when $\alpha>1$ in general Hilbert spaces. In this work, we revisit this phenomenon from a sparse learning perspective. We study realizable regression problems with controlled feature coherence and derive explicit lower bounds on the residual norm, showing that over-decaying step-size schedules induce structural stagnation even in low-dimensional sparse settings. Numerical experiments confirm the theoretical predictions and illustrate the role of feature coherence. Our results provide insight into step-size design in greedy sparse learning.