A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification

TL;DR

This paper introduces MFM-Geom, a geometric multimodal foundation model that integrates bp-MRI and clinical reports using Riemannian deep learning, significantly improving prostate cancer classification accuracy with limited training data.

Contribution

The study presents a novel geometric multimodal foundation model that combines imaging and clinical data through Riemannian deep learning, enhancing robustness and performance in prostate cancer diagnosis.

Findings

MFM-Geom outperforms baseline models by 8.3% in AUC-PR.

Achieves an AUC-PR of 90.67% with limited data.

Demonstrates robustness on external datasets.

Abstract

Prostate cancer (PCa) is one of the most common cancers in men worldwide. Bi-parametric MRI (bp-MRI) and clinical variables are crucial for PCa identification and improving treatment decisions. However, this process is subjective to expert interpretations. Furthermore, most existing computer-aided diagnosis methods focus on imaging-based models, overlooking the clinical context and suffering from data scarcity, limiting their ability to learn robust representations. We propose a geometric multimodal Foundation Model (FM), named MFM-Geom, that learns representations from bp-MRI and clinical reports, encoding visual findings and information from the context of clinical variables. In the representations classification head, the approach leverages symmetric positive definite (SPD) matrices and Riemannian deep learning to integrate imaging-text representations from a biomedical multimodal FM. Using 10% of the training data, MFM-Geom outperformed baseline class token embedding-based classification (+8.3%, AUC-PR of 90.67). Generalization on external dataset confirmed the robustness of fine-tuning biomedical FM, achieving an AUC-PR of 90.6.