Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics

TL;DR

This paper introduces a hierarchical multimodal transformer that effectively integrates multi-resolution pathology images and genomics data for improved cancer survival prediction, reducing computational costs while enhancing accuracy.

Contribution

A novel hierarchical multimodal transformer framework that combines multi-resolution WSIs and genomics data for better survival prediction with lower resource requirements.

Findings

Achieved an average c-index of 0.673 across five cancer types.

Outperformed state-of-the-art multimodality methods.

Reduced GPU resource consumption compared to benchmarks.

Abstract

Learning good representation of giga-pixel level whole slide pathology images (WSI) for downstream tasks is critical. Previous studies employ multiple instance learning (MIL) to represent WSIs as bags of sampled patches because, for most occasions, only slide-level labels are available, and only a tiny region of the WSI is disease-positive area. However, WSI representation learning still remains an open problem due to: (1) patch sampling on a higher resolution may be incapable of depicting microenvironment information such as the relative position between the tumor cells and surrounding tissues, while patches at lower resolution lose the fine-grained detail; (2) extracting patches from giant WSI results in large bag size, which tremendously increases the computational cost. To solve the problems, this paper proposes a hierarchical-based multimodal transformer framework that learns a hierarchical mapping between pathology images and corresponding genes. Precisely, we randomly extract instant-level patch features from WSIs with different magnification. Then a co-attention mapping between imaging and genomics is learned to uncover the pairwise interaction and reduce the space complexity of imaging features. Such early fusion makes it computationally feasible to use MIL Transformer for the survival prediction task. Our architecture requires fewer GPU resources compared with benchmark methods while maintaining better WSI representation ability. We evaluate our approach on five cancer types from the Cancer Genome Atlas database and achieved an average c-index of $0.673$, outperforming the state-of-the-art multimodality methods.