A Conformal Predictive Measure for Assessing Catastrophic Forgetting

TL;DR

This paper introduces a conformal prediction-based metric called CPCF to effectively quantify and monitor catastrophic forgetting in continual learning, demonstrating strong correlation with task accuracy across benchmarks.

Contribution

The paper presents a novel conformal prediction-based metric for assessing catastrophic forgetting, offering a dynamic and interpretable measure suitable for real-world applications.

Findings

CPCF correlates strongly with task accuracy

The method is validated on four benchmark datasets

CPCF provides a reliable measure of forgetting

Abstract

This work introduces a novel methodology for assessing catastrophic forgetting (CF) in continual learning. We propose a new conformal prediction (CP)-based metric, termed the Conformal Prediction Confidence Factor (CPCF), to quantify and evaluate CF effectively. Our framework leverages adaptive CP to estimate forgetting by monitoring the model's confidence on previously learned tasks. This approach provides a dynamic and practical solution for monitoring and measuring CF of previous tasks as new ones are introduced, offering greater suitability for real-world applications. Experimental results on four benchmark datasets demonstrate a strong correlation between CPCF and the accuracy of previous tasks, validating the reliability and interpretability of the proposed metric. Our results highlight the potential of CPCF as a robust and effective tool for assessing and understanding CF in dynamic learning environments.