Hessian Eigenvectors and Principal Component Analysis of Neural Network Weight Matrices

TL;DR

This paper investigates the relationship between Hessian eigenvectors and neural network weights, revealing how principal component analysis and singular value decomposition can identify critical parameter directions to improve understanding and mitigate catastrophic forgetting.

Contribution

It introduces a novel analysis linking Hessian eigenvectors with network weights and demonstrates how PCA and SVD can identify important directions to address catastrophic forgetting.

Findings

Hessian eigenvectors correlate with network weights based on eigenvalue magnitude.

Principal component analysis effectively approximates the Hessian, especially using update parameters.

Deeper layers exhibit larger Hessian eigenvalues, indicating higher curvature.

Abstract

This study delves into the intricate dynamics of trained deep neural networks and their relationships with network parameters. Trained networks predominantly continue training in a single direction, known as the drift mode. This drift mode can be explained by the quadratic potential model of the loss function, suggesting a slow exponential decay towards the potential minima. We unveil a correlation between Hessian eigenvectors and network weights. This relationship, hinging on the magnitude of eigenvalues, allows us to discern parameter directions within the network. Notably, the significance of these directions relies on two defining attributes: the curvature of their potential wells (indicated by the magnitude of Hessian eigenvalues) and their alignment with the weight vectors. Our exploration extends to the decomposition of weight matrices through singular value decomposition. This approach proves practical in identifying critical directions within the Hessian, considering both their magnitude and curvature. Furthermore, our examination showcases the applicability of principal component analysis in approximating the Hessian, with update parameters emerging as a superior choice over weights for this purpose. Remarkably, our findings unveil a similarity between the largest Hessian eigenvalues of individual layers and the entire network. Notably, higher eigenvalues are concentrated more in deeper layers. Leveraging these insights, we venture into addressing catastrophic forgetting, a challenge of neural networks when learning new tasks while retaining knowledge from previous ones. By applying our discoveries, we formulate an effective strategy to mitigate catastrophic forgetting, offering a possible solution that can be applied to networks of varying scales, including larger architectures.