On the Computation of the Fisher Information in Continual Learning

TL;DR

This paper compares different implementations of Fisher Information in EWC for continual learning, revealing that many results could improve with better computation methods.

Contribution

It provides an empirical comparison of Fisher Information computation methods in EWC, highlighting the impact on continual learning performance.

Findings

Different Fisher Information implementations yield varying results.

Many existing EWC results may be suboptimal due to computation choices.

Improved Fisher computation can enhance continual learning outcomes.

Abstract

One of the most popular methods for continual learning with deep neural networks is Elastic Weight Consolidation (EWC), which involves computing the Fisher Information. The exact way in which the Fisher Information is computed is however rarely described, and multiple different implementations for it can be found online. This blog post discusses and empirically compares several often-used implementations, which highlights that many currently reported results for EWC could likely be improved by changing the way the Fisher Information is computed.