Selfless Sequential Learning

TL;DR

This paper introduces a novel regularizer based on neural inhibition to promote sparsity in neuron activations, enabling fixed-capacity models to better accommodate sequential tasks by leaving capacity for future learning.

Contribution

It proposes a new regularizer inspired by brain lateral inhibition that encourages local neuron activation sparsity, improving lifelong learning performance with fixed model capacity.

Findings

Regularizer increases neuron activation sparsity.

Enhanced sparsity improves performance on sequential learning tasks.

Combining the regularizer with existing methods yields consistent gains.

Abstract

Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for future tasks to be added and thus leave enough capacity for them. To achieve Selfless Sequential Learning we study different regularization strategies and activation functions. We find that imposing sparsity at the level of the representation (i.e.~neuron activations) is more beneficial for sequential learning than encouraging parameter sparsity. In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition. It results in few active neurons which in turn leaves more free neurons to be utilized by upcoming tasks. As neural inhibition over an entire layer can be too drastic, especially for complex tasks requiring strong representations, our regularizer only inhibits other neurons in a local neighbourhood, inspired by lateral inhibition processes in the brain. We combine our novel regularizer, with state-of-the-art lifelong learning methods that penalize changes to important previously learned parts of the network. We show that our new regularizer leads to increased sparsity which translates in consistent performance improvement %over alternative regularizers we studied on diverse datasets.