Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

TL;DR

This paper presents a scalable Bayesian online learning method using Kronecker factored Laplace approximations to mitigate catastrophic forgetting in neural networks, achieving high accuracy on permuted MNIST tasks.

Contribution

It introduces a novel Kronecker factored online Laplace approximation method that efficiently approximates the posterior in neural networks for continual learning.

Findings

Achieves over 90% test accuracy on permuted MNIST sequence.

Outperforms existing methods in overcoming catastrophic forgetting.

Scalable approach suitable for modern neural network architectures.

Abstract

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90% test accuracy across a sequence of 50 instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.