MEDFORM: A Foundation Model for Contrastive Learning of CT Imaging and Clinical Numeric Data in Multi-Cancer Analysis

TL;DR

MEDFORM is a novel multimodal foundation model that leverages contrastive learning to integrate CT imaging and clinical data, enhancing multi-cancer classification and few-shot learning capabilities.

Contribution

This study introduces MEDFORM, a new pre-training framework combining self-supervised and cross-modal contrastive learning for multimodal cancer analysis.

Findings

Improved cancer classification accuracy across multiple cancer types.

Robust performance in few-shot learning scenarios.

Effective integration of CT and clinical data through dual pre-training.

Abstract

Computed tomography (CT) and clinical numeric data are essential modalities for cancer evaluation, but building large-scale multimodal training datasets for developing medical foundation models remains challenging due to the structural complexity of multi-slice CT data and high cost of expert annotation. In this study, we propose MEDFORM, a multimodal pre-training strategy that guides CT image representation learning using complementary information from clinical data for medical foundation model development. MEDFORM efficiently processes CT slice through multiple instance learning (MIL) and adopts a dual pre-training strategy: first pretraining the CT slice feature extractor using SimCLR-based self-supervised learning, then aligning CT and clinical modalities through cross-modal contrastive learning. Our model was pre-trained on three different cancer types: lung cancer (141,171 slices), breast cancer (8,100 slices), colorectal cancer (10,393 slices). The experimental results demonstrated that this dual pre-training strategy improves cancer classification performance and maintains robust performance in few-shot learning scenarios. Code available at https://github.com/DigitalHealthcareLab/25MultiModalFoundationModel.git