MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality

TL;DR

MoRA is a novel low-rank adaptation method that enhances multi-modal disease diagnosis by efficiently handling missing modalities and reducing computational costs, outperforming existing techniques.

Contribution

Introduces MoRA, a low-rank, modality-aware adaptation method that improves multi-modal disease diagnosis with missing data and low resource requirements.

Findings

MoRA outperforms existing methods in disease diagnosis accuracy.

MoRA significantly reduces training parameters to less than 1.6%.

MoRA demonstrates robustness and efficiency in experimental results.

Abstract

Multi-modal pre-trained models efficiently extract and fuse features from different modalities with low memory requirements for fine-tuning. Despite this efficiency, their application in disease diagnosis is under-explored. A significant challenge is the frequent occurrence of missing modalities, which impairs performance. Additionally, fine-tuning the entire pre-trained model demands substantial computational resources. To address these issues, we introduce Modality-aware Low-Rank Adaptation (MoRA), a computationally efficient method. MoRA projects each input to a low intrinsic dimension but uses different modality-aware up-projections for modality-specific adaptation in cases of missing modalities. Practically, MoRA integrates into the first block of the model, significantly improving performance when a modality is missing. It requires minimal computational resources, with less than 1.6% of the trainable parameters needed compared to training the entire model. Experimental results show that MoRA outperforms existing techniques in disease diagnosis, demonstrating superior performance, robustness, and training efficiency.