Is forgetting less a good inductive bias for forward transfer?

TL;DR

This paper proposes a new way to measure forward transfer in continual learning, showing that less forgetful representations improve learning efficiency and are more diverse and discriminative.

Contribution

It introduces a novel measure of forward transfer unaffected by retention constraints, linking less forgetting to better transfer and representation quality.

Findings

Less forgetful representations enhance forward transfer.

Less forgetful representations are more diverse and discriminative.

A strong correlation exists between retention and learning efficiency.

Abstract

One of the main motivations of studying continual learning is that the problem setting allows a model to accrue knowledge from past tasks to learn new tasks more efficiently. However, recent studies suggest that the key metric that continual learning algorithms optimize, reduction in catastrophic forgetting, does not correlate well with the forward transfer of knowledge. We believe that the conclusion previous works reached is due to the way they measure forward transfer. We argue that the measure of forward transfer to a task should not be affected by the restrictions placed on the continual learner in order to preserve knowledge of previous tasks. Instead, forward transfer should be measured by how easy it is to learn a new task given a set of representations produced by continual learning on previous tasks. Under this notion of forward transfer, we evaluate different continual learning algorithms on a variety of image classification benchmarks. Our results indicate that less forgetful representations lead to a better forward transfer suggesting a strong correlation between retaining past information and learning efficiency on new tasks. Further, we found less forgetful representations to be more diverse and discriminative compared to their forgetful counterparts.