Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction

TL;DR

This paper introduces PathOmics, a multimodal transformer that integrates pathology images and genomics data for improved survival outcome prediction in colon cancer, emphasizing unsupervised pretraining and data efficiency.

Contribution

The study presents a novel multimodal transformer model with unsupervised pretraining for integrating pathology and genomics data in cancer survival prediction.

Findings

Outperforms state-of-the-art methods on TCGA colon and rectum cancer datasets.

Effective in data-limited scenarios for survival prediction.

Demonstrates the benefit of multimodal integration and unsupervised pretraining.

Abstract

Survival outcome assessment is challenging and inherently associated with multiple clinical factors (e.g., imaging and genomics biomarkers) in cancer. Enabling multimodal analytics promises to reveal novel predictive patterns of patient outcomes. In this study, we propose a multimodal transformer (PathOmics) integrating pathology and genomics insights into colon-related cancer survival prediction. We emphasize the unsupervised pretraining to capture the intrinsic interaction between tissue microenvironments in gigapixel whole slide images (WSIs) and a wide range of genomics data (e.g., mRNA-sequence, copy number variant, and methylation). After the multimodal knowledge aggregation in pretraining, our task-specific model finetuning could expand the scope of data utility applicable to both multi- and single-modal data (e.g., image- or genomics-only). We evaluate our approach on both TCGA colon and rectum cancer cohorts, showing that the proposed approach is competitive and outperforms state-of-the-art studies. Finally, our approach is desirable to utilize the limited number of finetuned samples towards data-efficient analytics for survival outcome prediction. The code is available at https://github.com/Cassie07/PathOmics.