Why is SAM Robust to Label Noise?

TL;DR

This paper investigates why Sharpness-Aware Minimization (SAM) is particularly effective under label noise, revealing that its robustness mainly stems from its influence on the network Jacobian rather than explicit logit weighting.

Contribution

The paper provides a theoretical analysis of SAM's robustness to label noise, highlighting the Jacobian effect as the key factor, and proposes cheaper alternatives that replicate its benefits.

Findings

SAM's robustness is primarily due to its effect on the network Jacobian.

Explicit logit weighting in SAM does not significantly impact performance.

Cheaper methods mimicking SAM's Jacobian effect achieve similar robustness.

Abstract

Sharpness-Aware Minimization (SAM) is most known for achieving state-of the-art performances on natural image and language tasks. However, its most pronounced improvements (of tens of percent) is rather in the presence of label noise. Understanding SAM's label noise robustness requires a departure from characterizing the robustness of minimas lying in "flatter" regions of the loss landscape. In particular, the peak performance under label noise occurs with early stopping, far before the loss converges. We decompose SAM's robustness into two effects: one induced by changes to the logit term and the other induced by changes to the network Jacobian. The first can be observed in linear logistic regression where SAM provably up-weights the gradient contribution from clean examples. Although this explicit up-weighting is also observable in neural networks, when we intervene and modify SAM to remove this effect, surprisingly, we see no visible degradation in performance. We infer that SAM's effect in deeper networks is instead explained entirely by the effect SAM has on the network Jacobian. We theoretically derive the implicit regularization induced by this Jacobian effect in two layer linear networks. Motivated by our analysis, we see that cheaper alternatives to SAM that explicitly induce these regularization effects largely recover the benefits in deep networks trained on real-world datasets.