Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization

TL;DR

This paper introduces Loss-Equated SAM (LE-SAM), a novel approach that focuses on loss-space budgets rather than fixed radii, leading to improved generalization and state-of-the-art results.

Contribution

LE-SAM redefines adversarial perturbation by inverting SAM's mechanism, emphasizing curvature over gradient norms for better optimization.

Findings

LE-SAM outperforms SAM and variants across multiple benchmarks.

LE-SAM achieves state-of-the-art generalization performance.

Extensive experiments validate the effectiveness of LE-SAM.

Abstract

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss within a fixed parameter-space radius neighborhood. SAM and its variants mainly rely on a first-order linearized surrogate, while flat minima are inherently a second-order (curvature) notion.We revisit this mismatch and propose Loss-Equated SAM (LE-SAM), which inverts the traditional SAM mechanism that fixed perturbation radius with a fixed loss-space budget,effectively removing gradient-norm-dominated learning signals and shifting optimization toward curvature-dominated terms. Extensive experiments across diverse benchmarks and tasks demonstrate the strong generalization ability of LESAM that consistently outperforms SAM and even its variants, achieving the state-of-the-art performance.