Unique Properties of Flat Minima in Deep Networks

TL;DR

This paper characterizes the properties of flat minima in deep networks, showing they lead to nearly balanced, coupled layer structures that preserve signal gain, with implications for understanding training dynamics.

Contribution

It provides a theoretical analysis of flat minima in linear neural networks, revealing their structure and coupling properties, and extends findings to nonlinear models.

Findings

Linear ResNets with zero initialization converge to the flattest minima.

Flat minima correspond to nearly balanced networks with stable layer gains.

Layers in flat minima are coupled through singular vectors, forming a dedicated signal path.

Abstract

It is well known that (stochastic) gradient descent has an implicit bias towards flat minima. In deep neural network training, this mechanism serves to screen out minima. However, the precise effect that this has on the trained network is not yet fully understood. In this paper, we characterize the flat minima in linear neural networks trained with a quadratic loss. First, we show that linear ResNets with zero initialization necessarily converge to the flattest of all minima. We then prove that these minima correspond to nearly balanced networks whereby the gain from the input to any intermediate representation does not change drastically from one layer to the next. Finally, we show that consecutive layers in flat minima solutions are coupled. That is, one of the left singular vectors of each weight matrix, equals one of the right singular vectors of the next matrix. This forms a distinct path from input to output, that, as we show, is dedicated to the signal that experiences the largest gain end-to-end. Experiments indicate that these properties are characteristic of both linear and nonlinear models trained in practice.