TMSS: An End-to-End Transformer-based Multimodal Network for Segmentation and Survival Prediction

TL;DR

This paper introduces TMSS, an end-to-end transformer-based multimodal network that mimics oncologists' fusion of medical images and patient data for improved cancer segmentation and survival prediction.

Contribution

The work presents a novel transformer-based model that integrates multimodal data in an end-to-end manner for simultaneous segmentation and prognosis tasks.

Findings

Outperforms state-of-the-art methods in survival prediction with a concordance index of 0.763

Achieves a dice score of 0.772 for segmentation, comparable to standalone models

Demonstrates effective multimodal data fusion in cancer prognosis and segmentation

Abstract

When oncologists estimate cancer patient survival, they rely on multimodal data. Even though some multimodal deep learning methods have been proposed in the literature, the majority rely on having two or more independent networks that share knowledge at a later stage in the overall model. On the other hand, oncologists do not do this in their analysis but rather fuse the information in their brain from multiple sources such as medical images and patient history. This work proposes a deep learning method that mimics oncologists' analytical behavior when quantifying cancer and estimating patient survival. We propose TMSS, an end-to-end Transformer based Multimodal network for Segmentation and Survival prediction that leverages the superiority of transformers that lies in their abilities to handle different modalities. The model was trained and validated for segmentation and prognosis tasks on the training dataset from the HEad & NeCK TumOR segmentation and the outcome prediction in PET/CT images challenge (HECKTOR). We show that the proposed prognostic model significantly outperforms state-of-the-art methods with a concordance index of 0.763+/-0.14 while achieving a comparable dice score of 0.772+/-0.030 to a standalone segmentation model. The code is publicly available.