Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation

TL;DR

This paper introduces a new continual learning setting with limited computational resources, emphasizing semi-supervised learning from sparse labels, and proposes DietCL, a method that effectively utilizes both labeled and unlabeled data to outperform existing approaches.

Contribution

The paper presents DietCL, a simple yet effective baseline for constrained continual learning that jointly leverages labeled and unlabeled data, significantly improving performance in sparse label and limited compute scenarios.

Findings

DietCL outperforms existing continual learning methods under constrained budgets.

It remains stable across various levels of label sparsity and computational constraints.

Extensive experiments validate the effectiveness of DietCL on multiple large-scale datasets.

Abstract

We propose and study a realistic Continual Learning (CL) setting where learning algorithms are granted a restricted computational budget per time step while training. We apply this setting to large-scale semi-supervised Continual Learning scenarios with sparse label rates. Previous proficient CL methods perform very poorly in this challenging setting. Overfitting to the sparse labeled data and insufficient computational budget are the two main culprits for such a poor performance. Our new setting encourages learning methods to effectively and efficiently utilize the unlabeled data during training. To that end, we propose a simple but highly effective baseline, DietCL, which utilizes both unlabeled and labeled data jointly. DietCL meticulously allocates computational budget for both types of data. We validate our baseline, at scale, on several datasets, e.g., CLOC, ImageNet10K, and CGLM, under constraint budget setups. DietCL outperforms, by a large margin, all existing supervised CL algorithms as well as more recent continual semi-supervised methods. Our extensive analysis and ablations demonstrate that DietCL is stable under a full spectrum of label sparsity, computational budget, and various other ablations.