Overshoot: Taking advantage of future gradients in momentum-based stochastic optimization

TL;DR

Overshoot is a new momentum-based stochastic optimization method that evaluates gradients at shifted weights to improve convergence speed and performance across various tasks.

Contribution

The paper introduces Overshoot, a novel gradient evaluation technique that enhances existing momentum optimizers by shifting weights in the momentum direction.

Findings

Overshoot achieves at least 15% reduction in optimization steps.

It outperforms standard and Nesterov's momentum on multiple tasks.

The method adds minimal computational overhead.

Abstract

Overshoot is a novel, momentum-based stochastic gradient descent optimization method designed to enhance performance beyond standard and Nesterov's momentum. In conventional momentum methods, gradients from previous steps are aggregated with the gradient at current model weights before taking a step and updating the model. Rather than calculating gradient at the current model weights, Overshoot calculates the gradient at model weights shifted in the direction of the current momentum. This sacrifices the immediate benefit of using the gradient w.r.t. the exact model weights now, in favor of evaluating at a point, which will likely be more relevant for future updates. We show that incorporating this principle into momentum-based optimizers (SGD with momentum and Adam) results in faster convergence (saving on average at least 15% of steps). Overshoot consistently outperforms both standard and Nesterov's momentum across a wide range of tasks and integrates into popular momentum-based optimizers with zero memory and small computational overhead.