Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep Learning

TL;DR

This study presents an interpretable multimodal deep learning model that integrates histology images and genomic data to improve prognosis prediction across multiple cancer types, revealing key morphological and molecular prognostic markers.

Contribution

The paper introduces a novel weakly-supervised multimodal deep learning approach that combines histology and genomics for joint prognosis prediction and interpretability across 14 cancer types.

Findings

Improved risk stratification in 9 out of 14 cancers using multimodal data.

Identification of tumor-infiltrating lymphocytes as favorable prognostic markers.

Development of an interactive database for biomarker discovery.

Abstract

The rapidly emerging field of deep learning-based computational pathology has demonstrated promise in developing objective prognostic models from histology whole slide images. However, most prognostic models are either based on histology or genomics alone and do not address how histology and genomics can be integrated to develop joint image-omic prognostic models. Additionally identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We used multimodal deep learning to integrate gigapixel whole slide pathology images, RNA-seq abundance, copy number variation, and mutation data from 5,720 patients across 14 major cancer types. Our interpretable, weakly-supervised, multimodal deep learning algorithm is able to fuse these heterogeneous modalities for predicting outcomes and discover prognostic features from these modalities that corroborate with poor and favorable outcomes via multimodal interpretability. We compared our model with unimodal deep learning models trained on histology slides and molecular profiles alone, and demonstrate performance increase in risk stratification on 9 out of 14 cancers. In addition, we analyze morphologic and molecular markers responsible for prognostic predictions across all cancer types. All analyzed data, including morphological and molecular correlates of patient prognosis across the 14 cancer types at a disease and patient level are presented in an interactive open-access database (http://pancancer.mahmoodlab.org) to allow for further exploration and prognostic biomarker discovery. To validate that these model explanations are prognostic, we further analyzed high attention morphological regions in WSIs, which indicates that tumor-infiltrating lymphocyte presence corroborates with favorable cancer prognosis on 9 out of 14 cancer types studied.