Negative eigenvalues of the Hessian in deep neural networks

TL;DR

This paper investigates the non-convex loss landscape of deep neural networks by analyzing Hessian eigenvalues, emphasizing the significance of negative eigenvalues and their impact on training dynamics.

Contribution

It provides a detailed analysis of the Hessian eigenvalues in deep networks and explores the effects of negative eigenvalues on optimization.

Findings

Negative eigenvalues are prevalent in deep network loss landscapes.

Handling negative eigenvalues can improve training stability.

The Hessian's eigendecomposition reveals insights into non-convexity.

Abstract

The loss function of deep networks is known to be non-convex but the precise nature of this nonconvexity is still an active area of research. In this work, we study the loss landscape of deep networks through the eigendecompositions of their Hessian matrix. In particular, we examine how important the negative eigenvalues are and the benefits one can observe in handling them appropriately.