A Universal Class of Sharpness-Aware Minimization Algorithms

TL;DR

This paper introduces a universal class of sharpness-aware minimization algorithms that can represent any function of the training loss Hessian, addressing limitations of previous measures and improving optimization for neural networks.

Contribution

The paper proposes a new, universal class of sharpness measures for optimization, with specific instances like Frob-SAM and Det-SAM, enabling more meaningful and invariant-aware minimization.

Findings

Universal expressiveness of the proposed sharpness measures

Explicit bias towards minimizing the new sharpness measures

Enhanced optimization performance demonstrated through experiments

Abstract

Recently, there has been a surge in interest in developing optimization algorithms for overparameterized models as achieving generalization is believed to require algorithms with suitable biases. This interest centers on minimizing sharpness of the original loss function; the Sharpness-Aware Minimization (SAM) algorithm has proven effective. However, most literature only considers a few sharpness measures, such as the maximum eigenvalue or trace of the training loss Hessian, which may not yield meaningful insights for non-convex optimization scenarios like neural networks. Additionally, many sharpness measures are sensitive to parameter invariances in neural networks, magnifying significantly under rescaling parameters. Motivated by these challenges, we introduce a new class of sharpness measures in this paper, leading to new sharpness-aware objective functions. We prove that these measures are \textit{universally expressive}, allowing any function of the training loss Hessian matrix to be represented by appropriate hyperparameters. Furthermore, we show that the proposed objective functions explicitly bias towards minimizing their corresponding sharpness measures, and how they allow meaningful applications to models with parameter invariances (such as scale-invariances). Finally, as instances of our proposed general framework, we present \textit{Frob-SAM} and \textit{Det-SAM}, which are specifically designed to minimize the Frobenius norm and the determinant of the Hessian of the training loss, respectively. We also demonstrate the advantages of our general framework through extensive experiments.