On the Maximum Hessian Eigenvalue and Generalization

TL;DR

This paper critically examines the relationship between the maximum Hessian eigenvalue ($_{max}$) and neural network generalization, revealing that smaller $_{max}$ does not always correlate with better generalization across various training interventions.

Contribution

The study provides empirical evidence challenging the assumption that $_{max}$ directly influences generalization, highlighting limitations of flatness-based metrics in explaining neural network performance.

Findings

Larger learning rates reduce $_{max}$ but do not always improve generalization at larger batch sizes.

Scaling batch size and learning rate can alter $_{max}$ without affecting generalization.

SAM reduces $_{max}$ but does not guarantee better generalization at larger batch sizes.

Abstract

The mechanisms by which certain training interventions, such as increasing learning rates and applying batch normalization, improve the generalization of deep networks remains a mystery. Prior works have speculated that "flatter" solutions generalize better than "sharper" solutions to unseen data, motivating several metrics for measuring flatness (particularly $\lambda_{max}$, the largest eigenvalue of the Hessian of the loss); and algorithms, such as Sharpness-Aware Minimization (SAM) [1], that directly optimize for flatness. Other works question the link between $\lambda_{max}$ and generalization. In this paper, we present findings that call $\lambda_{max}$'s influence on generalization further into question. We show that: (1) while larger learning rates reduce $\lambda_{max}$ for all batch sizes, generalization benefits sometimes vanish at larger batch sizes; (2) by scaling batch size and learning rate simultaneously, we can change $\lambda_{max}$ without affecting generalization; (3) while SAM produces smaller $\lambda_{max}$ for all batch sizes, generalization benefits (also) vanish with larger batch sizes; (4) for dropout, excessively high dropout probabilities can degrade generalization, even as they promote smaller $\lambda_{max}$; and (5) while batch-normalization does not consistently produce smaller $\lambda_{max}$, it nevertheless confers generalization benefits. While our experiments affirm the generalization benefits of large learning rates and SAM for minibatch SGD, the GD-SGD discrepancy demonstrates limits to $\lambda_{max}$'s ability to explain generalization in neural networks.