Don't Be So Positive: Negative Step Sizes in Second-Order Methods

TL;DR

This paper demonstrates that allowing negative step sizes in second-order methods, combined with Wolfe line search, enhances their effectiveness for neural network optimization, challenging the traditional avoidance of negative curvature information.

Contribution

The paper introduces the use of negative step sizes in second-order methods for neural networks, showing their global convergence and improved performance over existing Hessian modification techniques.

Findings

Negative step sizes improve optimization effectiveness.

Methods with negative step sizes are globally convergent.

Experimental results favor negative step sizes over traditional Hessian modifications.

Abstract

The value of second-order methods lies in the use of curvature information. Yet, this information is costly to extract and once obtained, valuable negative curvature information is often discarded so that the method is globally convergent. This limits the effectiveness of second-order methods in modern machine learning. In this paper, we show that second-order and second-order-like methods are promising optimizers for neural networks provided that we add one ingredient: negative step sizes. We show that under very general conditions, methods that produce ascent directions are globally convergent when combined with a Wolfe line search that allows both positive and negative step sizes. We experimentally demonstrate that using negative step sizes is often more effective than common Hessian modification methods.