Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training

TL;DR

This paper introduces Dimer-Enhanced Optimization (DEO), a physics-inspired first-order method that efficiently escapes saddle points in neural network training by approximating curvature without full Hessian computation.

Contribution

The paper proposes DEO, a novel first-order optimization framework that adapts the Dimer method to improve navigation of complex loss landscapes in neural networks.

Findings

DEO effectively escapes saddle points in neural network training.

Preliminary results show DEO improves training efficiency on a Transformer toy model.

DEO offers a computationally feasible way to incorporate curvature information in large-scale models.

Abstract

First-order optimization methods, such as SGD and Adam, are widely used for training large-scale deep neural networks due to their computational efficiency and robust performance. However, relying solely on gradient information, these methods often struggle to navigate complex loss landscapes with flat regions, plateaus, and saddle points. Second-order methods, which use curvature information from the Hessian matrix, can address these challenges but are computationally infeasible for large models. The Dimer method, a first-order technique that constructs two closely spaced points to probe the local geometry of a potential energy surface, efficiently estimates curvature using only gradient information. Inspired by its use in molecular dynamics simulations for locating saddle points, we propose Dimer-Enhanced Optimization (DEO), a novel framework to escape saddle points in neural network training. DEO adapts the Dimer method to explore a broader region of the loss landscape, approximating the Hessian's smallest eigenvector without computing the full matrix. By periodically projecting the gradient onto the subspace orthogonal to the minimum curvature direction, DEO guides the optimizer away from saddle points and flat regions, enhancing training efficiency with non-stepwise updates. Preliminary experiments on a Transformer toy model show DEO achieves competitive performance compared to standard first-order methods, improving navigation of complex loss landscapes. Our work repurposes physics-inspired, first-order curvature estimation to enhance neural network training in high-dimensional spaces.