Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling

TL;DR

This paper improves the convergence rates of forward gradient descent (FGD) by using repeated sampling, making it comparable to stochastic gradient descent (SGD) especially when the number of steps per sample is large, and it adapts to low-dimensional input structures.

Contribution

It introduces a method of repeated sampling in FGD that reduces the suboptimality factor from d to d/( ext{ or } d), improving convergence rates and adapting to low-dimensional structures.

Findings

FGD with  steps per sample matches SGD convergence when  ormation

Repeated sampling reduces the suboptimality factor from d to d/( ext{ or } d)

Method adapts to low-dimensional input structures

Abstract

Forward gradient descent (FGD) has been proposed as a biologically more plausible alternative of gradient descent as it can be computed without backward pass. Considering the linear model with $d$ parameters, previous work has found that the prediction error of FGD is, however, by a factor $d$ slower than the prediction error of stochastic gradient descent (SGD). In this paper we show that by computing $\ell$ FGD steps based on each training sample, this suboptimality factor becomes $d/(\ell \wedge d)$ and thus the suboptimality of the rate disappears if $\ell \gtrsim d.$ We also show that FGD with repeated sampling can adapt to low-dimensional structure in the input distribution. The main mathematical challenge lies in controlling the dependencies arising from the repeated sampling process.