Sharpness-Aware Minimization with Adaptive Regularization for Training Deep Neural Networks

TL;DR

This paper introduces SAMAR, an adaptive regularization method for sharpness-aware minimization that dynamically adjusts regularization based on model sharpness, improving generalization in deep neural network training.

Contribution

We propose SAMAR, a novel adaptive regularization technique for SAM, with theoretical convergence guarantees and demonstrated effectiveness on image recognition benchmarks.

Findings

SAMAR improves accuracy on CIFAR-10 and CIFAR-100

SAMAR enhances model generalization

Theoretical convergence of SAMAR is established

Abstract

Sharpness-Aware Minimization (SAM) has proven highly effective in improving model generalization in machine learning tasks. However, SAM employs a fixed hyperparameter associated with the regularization to characterize the sharpness of the model. Despite its success, research on adaptive regularization methods based on SAM remains scarce. In this paper, we propose the SAM with Adaptive Regularization (SAMAR), which introduces a flexible sharpness ratio rule to update the regularization parameter dynamically. We provide theoretical proof of the convergence of SAMAR for functions satisfying the Lipschitz continuity. Additionally, experiments on image recognition tasks using CIFAR-10 and CIFAR-100 demonstrate that SAMAR enhances accuracy and model generalization.