Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate

TL;DR

This paper proves the convergence of gap guided SAM algorithms with increasing batch sizes or decaying learning rates and demonstrates that these strategies find flatter minima, improving generalization in deep neural networks.

Contribution

It provides the first theoretical convergence analysis of GSAM with increasing batch size and decaying learning rate, and empirically shows their effectiveness in finding flatter minima.

Findings

Increasing batch size or decaying learning rate leads to flatter minima.

GSAM with increasing batch size converges theoretically.

Flatter minima correlate with better generalization.

Abstract

The sharpness-aware minimization (SAM) algorithm and its variants, including gap guided SAM (GSAM), have been successful at improving the generalization capability of deep neural network models by finding flat local minima of the empirical loss in training. Meanwhile, it has been shown theoretically and practically that increasing the batch size or decaying the learning rate avoids sharp local minima of the empirical loss. In this paper, we consider the GSAM algorithm with increasing batch sizes or decaying learning rates, such as cosine annealing or linear learning rate, and theoretically show its convergence. Moreover, we numerically compare SAM (GSAM) with and without an increasing batch size and conclude that using an increasing batch size or decaying learning rate finds flatter local minima than using a constant batch size and learning rate.