A Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methods

TL;DR

This paper introduces a learn-to-optimize method for coordinate-wise step sizes in quasi-Newton methods, specifically BFGS, using LSTM networks to improve convergence speed while maintaining theoretical guarantees.

Contribution

It provides a theoretical analysis for coordinate-wise step sizes in BFGS and develops a novel L2O approach that learns optimal step sizes with proven convergence properties.

Findings

Achieves up to 4x faster convergence than baseline methods.

Demonstrates effectiveness across diverse optimization tasks.

Provides theoretical conditions ensuring stability and convergence.

Abstract

Tuning step sizes is crucial for the stability and efficiency of optimization algorithms. While adaptive coordinate-wise step sizes have been shown to outperform scalar step size in first-order methods, their use in second-order methods is still under-explored and more challenging. Current approaches, including hypergradient descent and cutting plane methods, offer limited improvements or encounter difficulties in second-order contexts. To address these limitations, we first conduct a theoretical analysis within the Broyden-Fletcher-Goldfarb-Shanno (BFGS) framework, a prominent quasi-Newton method, and derive sufficient conditions for coordinate-wise step sizes that ensure convergence and stability. Building on this theoretical foundation, we introduce a novel learn-to-optimize (L2O) method that employs LSTM-based networks to learn optimal step sizes by leveraging insights from past optimization trajectories, while inherently respecting the derived theoretical guarantees. Extensive experiments demonstrate that our approach achieves substantial improvements over scalar step size methods and hypergradient descent-based method, offering up to 4$\times$ faster convergence across diverse optimization tasks.