Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

TL;DR

This paper investigates the asymmetry in the loss landscape of deep neural networks, revealing how factors like noise and activation functions influence valley symmetry, with implications for model fusion and federated learning.

Contribution

It systematically analyzes the causes of valley asymmetry in DNNs and introduces the concept of sign consistency as a key factor affecting model interpolation and alignment.

Findings

Sign consistency correlates with valley symmetry.

Imposing sign alignment improves federated learning.

Theoretical insights explain the asymmetry phenomenon.

Abstract

Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the {\it degree of sign consistency} between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.