Foundation Models as Class-Incremental Learners for Dermatological Image Classification

TL;DR

This paper evaluates the use of frozen foundation models for class-incremental learning in dermatology, demonstrating state-of-the-art results and exploring zero-training scenarios with prototype classifiers.

Contribution

It introduces a simple incremental learning approach with frozen foundation models and lightweight classifiers, achieving competitive performance without forgetting.

Findings

State-of-the-art performance in dermatological CIL

Effective zero-training with prototype classifiers

Frozen foundation models excel in continual learning

Abstract

Class-Incremental Learning (CIL) aims to learn new classes over time without forgetting previously acquired knowledge. The emergence of foundation models (FM) pretrained on large datasets presents new opportunities for CIL by offering rich, transferable representations. However, their potential for enabling incremental learning in dermatology remains largely unexplored. In this paper, we systematically evaluate frozen FMs pretrained on large-scale skin lesion datasets for CIL in dermatological disease classification. We propose a simple yet effective approach where the backbone remains frozen, and a lightweight MLP is trained incrementally for each task. This setup achieves state-of-the-art performance without forgetting, outperforming regularization, replay, and architecture based methods. To further explore the capabilities of frozen FMs, we examine zero training scenarios using nearest mean classifiers with prototypes derived from their embeddings. Through extensive ablation studies, we demonstrate that this prototype based variant can also achieve competitive results. Our findings highlight the strength of frozen FMs for continual learning in dermatology and support their broader adoption in real world medical applications. Our code and datasets are available here.