Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients

TL;DR

This paper explores multi-tangent forward gradients as an efficient alternative to backpropagation, showing that using multiple tangents enhances gradient approximation and optimization in neural network training.

Contribution

It introduces a novel method for combining multiple forward gradients using orthogonal projections, improving approximation accuracy and training performance.

Findings

Increasing tangents improves gradient approximation.

Multi-tangent approach enhances optimization performance.

Method is effective across various tasks.

Abstract

The gradients used to train neural networks are typically computed using backpropagation. While an efficient way to obtain exact gradients, backpropagation is computationally expensive, hinders parallelization, and is biologically implausible. Forward gradients are an approach to approximate the gradients from directional derivatives along random tangents computed by forward-mode automatic differentiation. So far, research has focused on using a single tangent per step. This paper provides an in-depth analysis of multi-tangent forward gradients and introduces an improved approach to combining the forward gradients from multiple tangents based on orthogonal projections. We demonstrate that increasing the number of tangents improves both approximation quality and optimization performance across various tasks.