The Online Patch Redundancy Eliminator (OPRE): A novel approach to online agnostic continual learning using dataset compression

TL;DR

This paper introduces OPRE, an online dataset compression method that enhances agnostic continual learning by reducing reliance on prior data assumptions, demonstrating superior performance on CIFAR datasets.

Contribution

The paper presents OPRE, a novel online dataset compression algorithm that improves agnostic continual learning without requiring extensive prior data assumptions.

Findings

OPRE outperforms several state-of-the-art online continual learning methods on CIFAR-10 and CIFAR-100.

OPRE requires minimal and interpretable assumptions about future data.

The approach suggests dataset compression is key to achieving fully agnostic continual learning.

Abstract

In order to achieve Continual Learning (CL), the problem of catastrophic forgetting, one that has plagued neural networks since their inception, must be overcome. The evaluation of continual learning methods relies on splitting a known homogeneous dataset and learning the associated tasks one after the other. We argue that most CL methods introduce a priori information about the data to come and cannot be considered agnostic. We exemplify this point with the case of methods relying on pretrained feature extractors, which are still used in CL. After showing that pretrained feature extractors imply a loss of generality with respect to the data that can be learned by the model, we then discuss other kinds of a priori information introduced in other CL methods. We then present the Online Patch Redundancy Eliminator (OPRE), an online dataset compression algorithm, which, along with the training of a classifier at test time, yields performance on CIFAR-10 and CIFAR-100 superior to a number of other state-of-the-art online continual learning methods. Additionally, OPRE requires only minimal and interpretable hypothesis on the data to come. We suggest that online dataset compression could well be necessary to achieve fully agnostic CL.