Augmenting Continual Learning of Diseases with LLM-Generated Visual Concepts

TL;DR

This paper introduces a novel continual learning framework for medical image classification that leverages LLM-generated visual concepts and cross-modal attention to improve semantic understanding and performance.

Contribution

It proposes a dynamic visual concept pool and a cross-modal attention module to enhance continual learning with richer semantic guidance.

Findings

Achieves state-of-the-art performance on medical and natural image datasets.

Effectively leverages LLM-generated visual concepts for improved classification.

Demonstrates the superiority of semantic-guided continual learning methods.

Abstract

Continual learning is essential for medical image classification systems to adapt to dynamically evolving clinical environments. The integration of multimodal information can significantly enhance continual learning of image classes. However, while existing approaches do utilize textual modality information, they solely rely on simplistic templates with a class name, thereby neglecting richer semantic information. To address these limitations, we propose a novel framework that harnesses visual concepts generated by large language models (LLMs) as discriminative semantic guidance. Our method dynamically constructs a visual concept pool with a similarity-based filtering mechanism to prevent redundancy. Then, to integrate the concepts into the continual learning process, we employ a cross-modal image-concept attention module, coupled with an attention loss. Through attention, the module can leverage the semantic knowledge from relevant visual concepts and produce class-representative fused features for classification. Experiments on medical and natural image datasets show our method achieves state-of-the-art performance, demonstrating the effectiveness and superiority of our method. We will release the code publicly.