Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks

TL;DR

This paper explores the complex geometry of deep neural network loss landscapes by categorizing 1D curves, developing algorithms to mine complex perturbation directions, visualizing diverse 2D surfaces, and providing theoretical insights via the Hessian matrix.

Contribution

It introduces a systematic categorization of 1D loss landscape curves, proposes algorithms to mine complex perturbation directions, and visualizes diverse 2D surface structures with theoretical explanations.

Findings

Loss landscapes along Gaussian noise directions are nearly basin-shaped.

New algorithms enable visualization of complex 1D and 2D loss surface structures.

Theoretical analysis links loss landscape features to Hessian matrix properties.

Abstract

The loss landscape of deep neural networks (DNNs) is commonly considered complex and wildly fluctuated. However, an interesting observation is that the loss surfaces plotted along Gaussian noise directions are almost v-basin ones with the perturbed model lying on the basin. This motivates us to rethink whether the 1D or 2D subspace could cover more complex local geometry structures, and how to mine the corresponding perturbation directions. This paper systematically and gradually categorizes the 1D curves from simple to complex, including v-basin, v-side, w-basin, w-peak, and vvv-basin curves. Notably, the latter two types are already hard to obtain via the intuitive construction of specific perturbation directions, and we need to propose proper mining algorithms to plot the corresponding 1D curves. Combining these 1D directions, various types of 2D surfaces are visualized such as the saddle surfaces and the bottom of a bottle of wine that are only shown by demo functions in previous works. Finally, we propose theoretical insights from the lens of the Hessian matrix to explain the observed several interesting phenomena.