Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning

TL;DR

This paper introduces MSLoRA-CR, a novel method for multimodal biomedical image incremental learning that enhances knowledge sharing and differentiation across modalities using contrastive regularization and LoRA modules, improving performance and efficiency.

Contribution

The paper proposes MSLoRA-CR, a new approach that fine-tunes modality-specific LoRA modules with contrastive regularization for effective incremental learning across biomedical image modalities.

Findings

MSLoRA-CR outperforms state-of-the-art methods in biomedical image incremental learning.

The approach achieves a 1.88% performance improvement over unconstrained incremental learning.

MSLoRA-CR maintains computational efficiency while enhancing knowledge transfer.

Abstract

Multimodal Biomedical Image Incremental Learning (MBIIL) is essential for handling diverse tasks and modalities in the biomedical domain, as training separate models for each modality or task significantly increases inference costs. Existing incremental learning methods focus on task expansion within a single modality, whereas MBIIL seeks to train a unified model incrementally across modalities. The MBIIL faces two challenges: I) How to preserve previously learned knowledge during incremental updates? II) How to effectively leverage knowledge acquired from existing modalities to support new modalities? To address these challenges, we propose MSLoRA-CR, a method that fine-tunes Modality-Specific LoRA modules while incorporating Contrastive Regularization to enhance intra-modality knowledge sharing and promote inter-modality knowledge differentiation. Our approach builds upon a large vision-language model (LVLM), keeping the pretrained model frozen while incrementally adapting new LoRA modules for each modality or task. Experiments on the incremental learning of biomedical images demonstrate that MSLoRA-CR outperforms both the state-of-the-art (SOTA) approach of training separate models for each modality and the general incremental learning method (incrementally fine-tuning LoRA). Specifically, MSLoRA-CR achieves a 1.88% improvement in overall performance compared to unconstrained incremental learning methods while maintaining computational efficiency. Our code is publicly available at https://github.com/VentusAislant/MSLoRA_CR.