CLIMD: A Curriculum Learning Framework for Imbalanced Multimodal Diagnosis

TL;DR

CLIMD introduces a curriculum learning framework tailored for imbalanced multimodal medical diagnosis, effectively improving model performance by focusing on key samples and gradually adapting to complex class distributions, outperforming existing methods.

Contribution

The paper proposes a novel curriculum learning framework that addresses class imbalance in multimodal medical diagnosis by combining intra-modal confidence and inter-modal complementarity, with a class distribution-guided scheduler.

Findings

Outperforms state-of-the-art methods on multiple datasets

Effectively handles imbalanced multimodal medical data

Easily integrated into existing models

Abstract

Clinicians usually combine information from multiple sources to achieve the most accurate diagnosis, and this has sparked increasing interest in leveraging multimodal deep learning for diagnosis. However, in real clinical scenarios, due to differences in incidence rates, multimodal medical data commonly face the issue of class imbalance, which makes it difficult to adequately learn the features of minority classes. Most existing methods tackle this issue with resampling or loss reweighting, but they are prone to overfitting or underfitting and fail to capture cross-modal interactions. Therefore, we propose a Curriculum Learning framework for Imbalanced Multimodal Diagnosis (CLIMD). Specifically, we first design multimodal curriculum measurer that combines two indicators, intra-modal confidence and inter-modal complementarity, to enable the model to focus on key samples and gradually adapt to complex category distributions. Additionally, a class distribution-guided training scheduler is introduced, which enables the model to progressively adapt to the imbalanced class distribution during training. Extensive experiments on multiple multimodal medical datasets demonstrate that the proposed method outperforms state-of-the-art approaches across various metrics and excels in handling imbalanced multimodal medical data. Furthermore, as a plug-and-play CL framework, CLIMD can be easily integrated into other models, offering a promising path for improving multimodal disease diagnosis accuracy. Code is publicly available at https://github.com/KHan-UJS/CLIMD.