Greedy Criterion in Orthogonal Greedy Learning

TL;DR

This paper introduces a new greedy criterion for orthogonal greedy learning that achieves near-optimal learning rates and better computational efficiency compared to traditional methods.

Contribution

It proposes the $\delta$-greedy threshold as an alternative to steepest gradient descent, along with an adaptive termination rule for improved learning performance.

Findings

Achieves almost optimal learning rate

Requires less computation than traditional OGL

Maintains near-optimal generalization performance

Abstract

Orthogonal greedy learning (OGL) is a stepwise learning scheme that starts with selecting a new atom from a specified dictionary via the steepest gradient descent (SGD) and then builds the estimator through orthogonal projection. In this paper, we find that SGD is not the unique greedy criterion and introduce a new greedy criterion, called "$\delta$-greedy threshold" for learning. Based on the new greedy criterion, we derive an adaptive termination rule for OGL. Our theoretical study shows that the new learning scheme can achieve the existing (almost) optimal learning rate of OGL. Plenty of numerical experiments are provided to support that the new scheme can achieve almost optimal generalization performance, while requiring less computation than OGL.