Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm

TL;DR

This paper unifies various sharpness-aware minimization methods using preconditioning, provides convergence analysis, and introduces infoSAM, a new algorithm that improves robustness and generalization in deep learning.

Contribution

It offers a unifying framework for SAM variants through preconditioning and proposes infoSAM, a novel algorithm addressing adversarial degradation issues.

Findings

infoSAM outperforms existing SAM variants on multiple benchmarks

Theoretical analysis confirms convergence properties of the unified approach

Preconditioning enhances the effectiveness of sharpness-aware optimization methods

Abstract

Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks.