Cross-Modality Masked Learning for Survival Prediction in ICI Treated NSCLC Patients

TL;DR

This paper introduces a novel multi-modal fusion framework using cross-modality masked learning for improved survival prediction in NSCLC patients treated with immunotherapy, leveraging a large dataset of CT images and clinical data.

Contribution

It presents a new cross-modality masked learning approach with specialized transformers for CT and clinical data, enhancing multi-modal feature integration for prognosis.

Findings

Outperforms existing methods in survival prediction accuracy.

Sets a new benchmark for multi-modal prognostic models.

Demonstrates effective fusion of imaging and clinical data.

Abstract

Accurate prognosis of non-small cell lung cancer (NSCLC) patients undergoing immunotherapy is essential for personalized treatment planning, enabling informed patient decisions, and improving both treatment outcomes and quality of life. However, the lack of large, relevant datasets and effective multi-modal feature fusion strategies pose significant challenges in this domain. To address these challenges, we present a large-scale dataset and introduce a novel framework for multi-modal feature fusion aimed at enhancing the accuracy of survival prediction. The dataset comprises 3D CT images and corresponding clinical records from NSCLC patients treated with immune checkpoint inhibitors (ICI), along with progression-free survival (PFS) and overall survival (OS) data. We further propose a cross-modality masked learning approach for medical feature fusion, consisting of two distinct branches, each tailored to its respective modality: a Slice-Depth Transformer for extracting 3D features from CT images and a graph-based Transformer for learning node features and relationships among clinical variables in tabular data. The fusion process is guided by a masked modality learning strategy, wherein the model utilizes the intact modality to reconstruct missing components. This mechanism improves the integration of modality-specific features, fostering more effective inter-modality relationships and feature interactions. Our approach demonstrates superior performance in multi-modal integration for NSCLC survival prediction, surpassing existing methods and setting a new benchmark for prognostic models in this context.