Adaptive Optimization with Examplewise Gradients

TL;DR

This paper introduces a generalized stochastic optimization framework that uses multiple gradient estimates per iteration, leading to a new optimizer called Eve, which shows slight improvements over Adam in initial tests.

Contribution

The paper proposes a new framework for stochastic optimization that incorporates multiple gradient estimates per iteration, and develops the Eve optimizer based on this approach.

Findings

Eve slightly outperforms Adam on small benchmarks.

Eve performs similarly or worse on larger benchmarks.

Further refinement and hyperparameter tuning are needed.

Abstract

We propose a new, more general approach to the design of stochastic gradient-based optimization methods for machine learning. In this new framework, optimizers assume access to a batch of gradient estimates per iteration, rather than a single estimate. This better reflects the information that is actually available in typical machine learning setups. To demonstrate the usefulness of this generalized approach, we develop Eve, an adaptation of the Adam optimizer which uses examplewise gradients to obtain more accurate second-moment estimates. We provide preliminary experiments, without hyperparameter tuning, which show that the new optimizer slightly outperforms Adam on a small scale benchmark and performs the same or worse on larger scale benchmarks. Further work is needed to refine the algorithm and tune hyperparameters.