SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation

TL;DR

SASSHA is a novel second-order optimization method that improves generalization in deep learning by explicitly reducing sharpness of minima and stabilizing Hessian approximations, achieving better performance than existing methods.

Contribution

This work introduces SASSHA, a second-order optimizer that enhances generalization by minimizing sharpness and stabilizing Hessian updates, with efficiency considerations.

Findings

SASSHA outperforms other methods in generalization across various deep learning tasks.

It demonstrates comparable or superior convergence and robustness.

SASSHA maintains efficiency with lazy Hessian updates.

Abstract

Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. In this work, we look into this issue through the lens of the loss landscape and find that existing second-order methods tend to converge to sharper minima compared to SGD. In response, we propose Sassha, a novel second-order method designed to enhance generalization by explicitly reducing sharpness of the solution, while stabilizing the computation of approximate Hessians along the optimization trajectory. In fact, this sharpness minimization scheme is crafted also to accommodate lazy Hessian updates, so as to secure efficiency besides flatness. To validate its effectiveness, we conduct a wide range of standard deep learning experiments where Sassha demonstrates its outstanding generalization performance that is comparable to, and mostly better than, other methods. We provide a comprehensive set of analyses including convergence, robustness, stability, efficiency, and cost.