Asymmetric Valleys: Beyond Sharp and Flat Local Minima

TL;DR

This paper introduces the concept of asymmetric valleys in deep neural network loss landscapes, showing that solutions biased towards flatter sides generalize better and are implicitly found by weight averaging during SGD, with batch normalization influencing this asymmetry.

Contribution

It formally defines asymmetric valleys, proves their impact on generalization, and links weight averaging and batch normalization to the emergence of such solutions.

Findings

Asymmetric valleys are common in deep networks' loss landscapes.

Biasing solutions towards the flat side improves generalization.

Weight averaging implicitly finds solutions in asymmetric valleys.

Abstract

Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good generalization performance. In this paper, we observe that local minima of modern deep networks are more than being flat or sharp. Specifically, at a local minimum there exist many asymmetric directions such that the loss increases abruptly along one side, and slowly along the opposite side--we formally define such minima as asymmetric valleys. Under mild assumptions, we prove that for asymmetric valleys, a solution biased towards the flat side generalizes better than the exact minimizer. Further, we show that simply averaging the weights along the SGD trajectory gives rise to such biased solutions implicitly. This provides a theoretical explanation for the intriguing phenomenon observed by Izmailov et al. (2018). In addition, we empirically find that batch normalization (BN) appears to be a major cause for asymmetric valleys.