Simple Linear Neuron Boosting

TL;DR

This paper introduces a novel neuron optimization method called Boosted Backpropagation, which reparameterizes networks for faster convergence and is applicable to various architectures like CNNs and transformers.

Contribution

It presents a new online, matrix-free algorithm for neuron optimization in function space, improving training speed and convergence across different network types.

Findings

Fast convergence in epochs and wall clock time

Applicable to convolutional networks and transformers

Simple implementation for local and distributed training

Abstract

Given a differentiable network architecture and loss function, we revisit optimizing the network's neurons in function space using Boosted Backpropagation (Grubb & Bagnell, 2010), in contrast to optimizing in parameter space. From this perspective, we reduce descent in the space of linear functions that optimizes the network's backpropagated-errors to a preconditioned gradient descent algorithm. We show that this preconditioned update rule is equivalent to reparameterizing the network to whiten each neuron's features, with the benefit that the normalization occurs outside of inference. In practice, we use this equivalence to construct an online estimator for approximating the preconditioner and we propose an online, matrix-free learning algorithm with adaptive step sizes. The algorithm is applicable whenever autodifferentiation is available, including convolutional networks and transformers, and it is simple to implement for both the local and distributed training settings. We demonstrate fast convergence both in terms of epochs and wall clock time on a variety of tasks and networks.