Revisiting Random Weight Perturbation for Efficiently Improving Generalization

TL;DR

This paper revisits random weight perturbation (RWP) for deep neural networks, proposing improvements that enhance its efficiency and performance, making it competitive with sharpness-aware minimization (SAM) in generalization tasks.

Contribution

The paper introduces improved RWP methods that better balance generalization and convergence, and optimize perturbation generation, achieving superior efficiency and comparable or better results than SAM.

Findings

Enhanced RWP methods outperform traditional RWP in large-scale problems.

Proposed techniques achieve similar or better generalization than SAM.

Improvements lead to more efficient training with better generalization.

Abstract

Improving the generalization ability of modern deep neural networks (DNNs) is a fundamental challenge in machine learning. Two branches of methods have been proposed to seek flat minima and improve generalization: one led by sharpness-aware minimization (SAM) minimizes the worst-case neighborhood loss through adversarial weight perturbation (AWP), and the other minimizes the expected Bayes objective with random weight perturbation (RWP). While RWP offers advantages in computation and is closely linked to AWP on a mathematical basis, its empirical performance has consistently lagged behind that of AWP. In this paper, we revisit the use of RWP for improving generalization and propose improvements from two perspectives: i) the trade-off between generalization and convergence and ii) the random perturbation generation. Through extensive experimental evaluations, we demonstrate that our enhanced RWP methods achieve greater efficiency in enhancing generalization, particularly in large-scale problems, while also offering comparable or even superior performance to SAM. The code is released at https://github.com/nblt/mARWP.