Is Multi-Task Learning an Upper Bound for Continual Learning?

TL;DR

This paper challenges the assumption that multi-task learning sets an upper bound for continual learning, demonstrating that in certain settings, continual learning can outperform multi-task learning on standard datasets.

Contribution

It introduces a novel continual self-supervised learning framework and provides empirical evidence that continual learning can surpass multi-task learning in specific scenarios.

Findings

Continual learning often outperforms multi-task learning on benchmark datasets.

A new continual self-supervised learning setting is proposed.

Empirical results on MNIST, CIFAR-10, and CIFAR-100 support the findings.

Abstract

Continual and multi-task learning are common machine learning approaches to learning from multiple tasks. The existing works in the literature often assume multi-task learning as a sensible performance upper bound for various continual learning algorithms. While this assumption is empirically verified for different continual learning benchmarks, it is not rigorously justified. Moreover, it is imaginable that when learning from multiple tasks, a small subset of these tasks could behave as adversarial tasks reducing the overall learning performance in a multi-task setting. In contrast, continual learning approaches can avoid the performance drop caused by such adversarial tasks to preserve their performance on the rest of the tasks, leading to better performance than a multi-task learner. This paper proposes a novel continual self-supervised learning setting, where each task corresponds to learning an invariant representation for a specific class of data augmentations. In this setting, we show that continual learning often beats multi-task learning on various benchmark datasets, including MNIST, CIFAR-10, and CIFAR-100.