CLOFAI: A Dataset of Real And Fake Image Classification Tasks for Continual Learning

TL;DR

This paper introduces CLOFAI, a new dataset for evaluating continual learning methods in classifying real versus fake images, highlighting the challenges and potential solutions for models that must adapt to new generative AI outputs without retraining from scratch.

Contribution

The paper presents CLOFAI, a novel dataset for continual learning in fake image detection, and benchmarks foundational methods, revealing their strengths and weaknesses in this context.

Findings

GEM and Experience Replay outperform EWC in this task.

EWC performs poorly compared to other methods.

The dataset provides a new benchmark for continual learning in image classification.

Abstract

The rapid advancement of generative AI models capable of creating realistic media has led to a need for classifiers that can accurately distinguish between genuine and artificially-generated images. A significant challenge for these classifiers emerges when they encounter images from generative models that are not represented in their training data, usually resulting in diminished performance. A typical approach is to periodically update the classifier's training data with images from the new generative models then retrain the classifier on the updated dataset. However, in some real-life scenarios, storage, computational, or privacy constraints render this approach impractical. Additionally, models used in security applications may be required to rapidly adapt. In these circumstances, continual learning provides a promising alternative, as the classifier can be updated without retraining on the entire dataset. In this paper, we introduce a new dataset called CLOFAI (Continual Learning On Fake and Authentic Images), which takes the form of a domain-incremental image classification problem. Moreover, we showcase the applicability of this dataset as a benchmark for evaluating continual learning methodologies. In doing this, we set a baseline on our novel dataset using three foundational continual learning methods -- EWC, GEM, and Experience Replay -- and find that EWC performs poorly, while GEM and Experience Replay show promise, performing significantly better than a Naive baseline. The dataset and code to run the experiments can be accessed from the following GitHub repository: https://github.com/Will-Doherty/CLOFAI.