Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics

TL;DR

This paper introduces a flexible, interpretable multimodal framework that effectively integrates whole slide images and incomplete genomics data for improved precision oncology, addressing heterogeneity and missing data issues.

Contribution

It presents a novel multimodal prototyping method with biological, alignment, fusion, and imputation components, handling incomplete data and enhancing interpretability.

Findings

Outperforms state-of-the-art methods on multiple tasks

Demonstrates robustness with incomplete genomics data

Provides interpretable insights into multimodal integration

Abstract

Multimodal approaches that integrate histology and genomics hold strong potential for precision oncology. However, phenotypic and genotypic heterogeneity limits the quality of intra-modal representations and hinders effective inter-modal integration. Furthermore, most existing methods overlook real-world clinical scenarios where genomics may be partially missing or entirely unavailable. We propose a flexible multimodal prototyping framework to integrate whole slide images and incomplete genomics for precision oncology. Our approach has four key components: 1) Biological Prototyping using text prompting and prototype-wise weighting; 2) Multiview Alignment through sample- and distribution-wise alignments; 3) Bipartite Fusion to capture both shared and modality-specific information for multimodal fusion; and 4) Semantic Genomics Imputation to handle missing data. Extensive experiments demonstrate the consistent superiority of the proposed method compared to other state-of-the-art approaches on multiple downstream tasks. The code is available at https://github.com/helenypzhang/Interpretable-Multimodal-Prototyping.