Bayesian Optimized Continual Learning with Attention Mechanism

TL;DR

This paper introduces BOCL, a continual learning model that dynamically expands neural networks using Bayesian optimization and attention mechanisms to better learn new tasks while avoiding forgetting.

Contribution

It presents a novel continual learning approach combining Bayesian optimization and attention to adapt network capacity and reuse knowledge effectively.

Findings

Outperforms state-of-the-art in preventing catastrophic forgetting

Better fitting of new tasks on MNIST and CIFAR-100 variants

Demonstrates effective dynamic network expansion

Abstract

Though neural networks have achieved much progress in various applications, it is still highly challenging for them to learn from a continuous stream of tasks without forgetting. Continual learning, a new learning paradigm, aims to solve this issue. In this work, we propose a new model for continual learning, called Bayesian Optimized Continual Learning with Attention Mechanism (BOCL) that dynamically expands the network capacity upon the arrival of new tasks by Bayesian optimization and selectively utilizes previous knowledge (e.g. feature maps of previous tasks) via attention mechanism. Our experiments on variants of MNIST and CIFAR-100 demonstrate that our methods outperform the state-of-the-art in preventing catastrophic forgetting and fitting new tasks better.