Normalization Layers Are All That Sharpness-Aware Minimization Needs

TL;DR

This paper demonstrates that perturbing only normalization layer parameters in sharpness-aware minimization (SAM) can outperform perturbing all parameters, suggesting normalization layers are key to SAM's effectiveness in improving generalization.

Contribution

The study reveals that focusing perturbations on normalization layers alone is sufficient for SAM's success, challenging the belief that reduced sharpness is the sole factor.

Findings

Perturbing only normalization parameters in SAM outperforms full-parameter perturbation.

Alternative sparse perturbations do not match normalization layer effectiveness.

Normalization layers are uniquely influential in SAM's performance.

Abstract

Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima and has been shown to enhance generalization performance in various settings. In this work we show that perturbing only the affine normalization parameters (typically comprising 0.1% of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters.This finding generalizes to different SAM variants and both ResNet (Batch Normalization) and Vision Transformer (Layer Normalization) architectures. We consider alternative sparse perturbation approaches and find that these do not achieve similar performance enhancement at such extreme sparsity levels, showing that this behaviour is unique to the normalization layers. Although our findings reaffirm the effectiveness of SAM in improving generalization performance, they cast doubt on whether this is solely caused by reduced sharpness.