Randomized Greedy Algorithms for Neural Network Optimization

TL;DR

This paper extends the orthogonal greedy algorithm's analysis to convex optimization, introduces a randomized discretization approach to reduce computational costs, and demonstrates its effectiveness through numerical experiments.

Contribution

It generalizes the optimal convergence rate of OGA to convex problems and proposes a practical randomized dictionary method for neural network training.

Findings

Proves the optimal convergence rate for the randomized greedy algorithm.

Shows significant reduction in dictionary size with high probability.

Validates the approach through experiments on PDEs and function approximation.

Abstract

Greedy algorithms have been successfully analyzed and applied in training neural networks for solving variational problems, ensuring guaranteed convergence orders. In this paper, we extend the analysis of the orthogonal greedy algorithm (OGA) to convex optimization problems, establishing its optimal convergence rate. This result broadens the applicability of OGA by generalizing its optimal convergence rate from function approximation to convex optimization problems. In addition, we also address the issue regarding practical applicability of greedy algorithms, which is due to significant computational costs from the subproblems that involve an exhaustive search over a discrete dictionary. We propose to use a more practical approach of randomly discretizing the dictionary at each iteration of the greedy algorithm. We quantify the required size of the randomized discrete dictionary and prove that, with high probability, the proposed algorithm realizes a weak greedy algorithm, achieving optimal convergence orders. Through numerous numerical experiments on function approximation, linear and nonlinear elliptic partial differential equations, we validate our analysis on the optimal convergence rate and demonstrate the advantage of using randomized discrete dictionaries over a deterministic one by showing orders of magnitude reductions in the size of the discrete dictionary, particularly in higher dimensions.