Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting

TL;DR

This paper investigates how different methods for calculating neural network weight importance correlate across layers, revealing strong and variable correlations and questioning their effectiveness in preventing catastrophic forgetting.

Contribution

It provides a layer-by-layer correlation analysis of various importance calculation methods used in EWC, offering explanations for their similarities and differences.

Findings

Some importance methods show strong correlation across layers.

Correlation varies from positive to negative depending on the layer.

Despite differences, all methods effectively help EWC mitigate forgetting.

Abstract

Following the invention in 2017 of the EWC method, several methods have been proposed to calculate the importance of neural network weights for use in the EWC method. Despite the significant difference in calculating the importance of weights, they all proved to be effective. Accordingly, a reasonable question arises as to how similar the importances of the weights calculated by different methods. To answer this question, we calculated layer-by-layer correlations of the importance of weights calculated by all those methods. As a result, it turned out that the importances of several of the methods correlated with each other quite strongly and we were able to present an explanation for such a correlation. At the same time, for other methods, the correlation can vary from strong on some layers of the network to negative on other layers. Which raises a reasonable question: why, despite the very different calculation methods, all those importances allow EWC method to overcome the catastrophic forgetting of neural networks perfectly?