SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization

TL;DR

SALR is a novel learning rate scheduler that adaptively adjusts the learning rate based on local sharpness, leading to better generalization and faster convergence in deep learning models.

Contribution

We introduce SALR, a sharpness-aware learning rate update method that automatically promotes flat minima, improving generalization and convergence across various models.

Findings

SALR enhances model generalization performance.

SALR accelerates convergence speed.

SALR finds flatter minima leading to better robustness.

Abstract

In an effort to improve generalization in deep learning and automate the process of learning rate scheduling, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.