Flat Minima and Generalization: Insights from Stochastic Convex Optimization

TL;DR

This paper investigates the relationship between flat minima and generalization in stochastic convex optimization, revealing that flat minima do not always guarantee good generalization and analyzing the limitations of sharpness-aware algorithms.

Contribution

It provides theoretical insights showing flat minima can still have poor generalization and analyzes the behavior of sharpness-aware algorithms like SA-GD and SAM.

Findings

Flat minima can have high population risk despite low empirical loss.

Sharpness-aware algorithms may converge to sharp minima with poor generalization.

Population risk bounds are derived for SA-GD and SAM using stability techniques.

Abstract

Understanding the generalization behavior of learning algorithms is a central goal of learning theory. A recently emerging explanation is that learning algorithms are successful in practice because they converge to flat minima, which have been consistently associated with improved generalization performance. In this work, we study the link between flat minima and generalization in the canonical setting of stochastic convex optimization with a non-negative, $\beta$-smooth objective. Our first finding is that, even in this fundamental and well-studied setting, flat empirical minima may incur trivial $\Omega(1)$ population risk while sharp minima generalizes optimally. Then, we show that this poor generalization behavior extends to two natural ''sharpness-aware'' algorithms originally proposed by Foret et al. (2021), designed to bias optimization toward flat solutions: Sharpness-Aware Gradient Descent (SA-GD) and Sharpness-Aware Minimization (SAM). For SA-GD, which performs gradient steps on the maximal loss in a predefined neighborhood, we prove that while it successfully converges to a flat minimum at a fast rate, the population risk of the solution can still be as large as $\Omega(1)$, indicating that even flat minima found algorithmically using a sharpness-aware gradient method might generalize poorly. For SAM, a computationally efficient approximation of SA-GD based on normalized ascent steps, we show that although it minimizes the empirical loss, it may converge to a sharp minimum and also incur population risk $\Omega(1)$. Finally, we establish population risk upper bounds for both SA-GD and SAM using algorithmic stability techniques.