Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning

TL;DR

This paper introduces a novel pre-training approach using generative models to simulate future data scenarios in continual learning, enhancing model robustness before actual deployment.

Contribution

It combines large language and image generation models to create synthetic datasets for pre-training, improving continual learning performance on fine-grained classification tasks.

Findings

Pre-training with synthetic data improves multiple CIL methods.

Synthetic datasets help learn useful representations despite domain gaps.

The approach enhances model preparedness for future data shifts.

Abstract

Continual learning requires a model to adapt to ongoing changes in the data distribution, and often to the set of tasks to be performed. It is rare, however, that the data and task changes are completely unpredictable. Given a description of an overarching goal or data theme, which we call a realm, humans can often guess what concepts are associated with it. We show here that the combination of a large language model and an image generation model can similarly provide useful premonitions as to how a continual learning challenge might develop over time. We use the large language model to generate text descriptions of semantically related classes that might potentially appear in the data stream in future. These descriptions are then rendered using Stable Diffusion to generate new labelled image samples. The resulting synthetic dataset is employed for supervised pre-training, but is discarded prior to commencing continual learning, along with the pre-training classification head. We find that the backbone of our pre-trained networks can learn representations useful for the downstream continual learning problem, thus becoming a valuable input to any existing continual learning method. Although there are complexities arising from the domain gap between real and synthetic images, we show that pre-training models in this manner improves multiple Class Incremenal Learning (CIL) methods on fine-grained image classification benchmarks. Supporting code can be found at https://github.com/cl-premonition/premonition.