Optimal Convergence Rates for the Orthogonal Greedy Algorithm

TL;DR

This paper establishes improved convergence rates for the orthogonal greedy algorithm when applied to dictionaries with convex hulls of small entropy, especially relevant for shallow neural networks, and confirms these rates empirically.

Contribution

It proves that convergence rates improve to O(n^{-rac{1}{2}-eta}) under entropy decay conditions, extending understanding of greedy algorithms in neural network contexts.

Findings

Convergence rates are improved for dictionaries with small entropy convex hulls.

Empirical results confirm the theoretical rates for shallow neural networks with Heaviside activation.

The improved rates are shown to be sharp, with a negative result on boundedness in variation norm.

Abstract

We analyze the orthogonal greedy algorithm when applied to dictionaries $\mathbb{D}$ whose convex hull has small entropy. We show that if the metric entropy of the convex hull of $\mathbb{D}$ decays at a rate of $O(n^{-\frac{1}{2}-\alpha})$ for $\alpha > 0$, then the orthogonal greedy algorithm converges at the same rate on the variation space of $\mathbb{D}$. This improves upon the well-known $O(n^{-\frac{1}{2}})$ convergence rate of the orthogonal greedy algorithm in many cases, most notably for dictionaries corresponding to shallow neural networks. These results hold under no additional assumptions on the dictionary beyond the decay rate of the entropy of its convex hull. In addition, they are robust to noise in the target function and can be extended to convergence rates on the interpolation spaces of the variation norm. We show empirically that the predicted rates are obtained for the dictionary corresponding to shallow neural networks with Heaviside activation function in two dimensions. Finally, we show that these improved rates are sharp and prove a negative result showing that the iterates generated by the orthogonal greedy algorithm cannot in general be bounded in the variation norm of $\mathbb{D}$.