Learning and approximation capability of orthogonal super greedy algorithm

TL;DR

This paper analyzes the approximation and learning capabilities of orthogonal super greedy algorithms (OSGA), demonstrating that they can reduce computational costs without sacrificing generalization performance when the dictionary is incoherent.

Contribution

It proves that OSGA maintains OGA's approximation ability for non-convex function classes with incoherent dictionaries and derives a tight generalization error bound for supervised learning.

Findings

OSGA reduces computational burden compared to OGA.

OSGA maintains approximation capability for non-convex classes.

A tight generalization error bound for OSGA is established.

Abstract

We consider the approximation capability of orthogonal super greedy algorithms (OSGA) and its applications in supervised learning. OSGA is concerned with selecting more than one atoms in each iteration step, which, of course, greatly reduces the computational burden when compared with the conventional orthogonal greedy algorithm (OGA). We prove that even for function classes that are not the convex hull of the dictionary, OSGA does not degrade the approximation capability of OGA provided the dictionary is incoherent. Based on this, we deduce a tight generalization error bound for OSGA learning. Our results show that in the realm of supervised learning, OSGA provides a possibility to further reduce the computational burden of OGA in the premise of maintaining its prominent generalization capability.