Adapter Learning in Pretrained Feature Extractor for Continual Learning of Diseases

TL;DR

This paper introduces ACL, an adapter-based continual learning framework that enables diagnosis systems to learn new diseases without forgetting old ones by using lightweight adapters and task-specific heads, demonstrated on medical image datasets.

Contribution

The paper proposes a novel ACL framework that employs lightweight adapters and a task-specific head to prevent catastrophic forgetting in disease diagnosis models during continual learning.

Findings

ACL outperforms existing methods on three medical image datasets.

The use of task-specific heads improves classification accuracy across tasks.

Lightweight adapters enable effective learning without altering the shared feature extractor.

Abstract

Currently intelligent diagnosis systems lack the ability of continually learning to diagnose new diseases once deployed, under the condition of preserving old disease knowledge. In particular, updating an intelligent diagnosis system with training data of new diseases would cause catastrophic forgetting of old disease knowledge. To address the catastrophic forgetting issue, an Adapter-based Continual Learning framework called ACL is proposed to help effectively learn a set of new diseases at each round (or task) of continual learning, without changing the shared feature extractor. The learnable lightweight task-specific adapter(s) can be flexibly designed (e.g., two convolutional layers) and then added to the pretrained and fixed feature extractor. Together with a specially designed task-specific head which absorbs all previously learned old diseases as a single "out-of-distribution" category, task-specific adapter(s) can help the pretrained feature extractor more effectively extract discriminative features between diseases. In addition, a simple yet effective fine-tuning is applied to collaboratively fine-tune multiple task-specific heads such that outputs from different heads are comparable and consequently the appropriate classifier head can be more accurately selected during model inference. Extensive empirical evaluations on three image datasets demonstrate the superior performance of ACL in continual learning of new diseases. The source code is available at https://github.com/GiantJun/CL_Pytorch.