CSF-Net: Cross-Modal Spatiotemporal Fusion Network for Pulmonary Nodule Malignancy Predicting

TL;DR

CSF-Net is a novel deep learning model that integrates follow-up CT scans and clinical data to improve the accuracy of pulmonary nodule malignancy prediction, mimicking clinical decision-making.

Contribution

The paper introduces CSF-Net, a cross-modal spatiotemporal fusion network that combines imaging and clinical data for better malignancy prediction of pulmonary nodules.

Findings

Achieved 89.74% accuracy on NLST-cmst dataset.

Outperformed existing methods in precision and AUC.

Demonstrated the effectiveness of multimodal data fusion.

Abstract

Pulmonary nodules are an early sign of lung cancer, and detecting them early is vital for improving patient survival rates. Most current methods use only single Computed Tomography (CT) images to assess nodule malignancy. However, doctors typically make a comprehensive assessment in clinical practice by integrating follow-up CT scans with clinical data. To enhance this process, our study introduces a Cross-Modal Spatiotemporal Fusion Network, named CSF-Net, designed to predict the malignancy of pulmonary nodules using follow-up CT scans. This approach simulates the decision-making process of clinicians who combine follow-up imaging with clinical information. CSF-Net comprises three key components: spatial feature extraction module, temporal residual fusion module, and cross-modal attention fusion module. Together, these modules enable precise predictions of nodule malignancy. Additionally, we utilized the publicly available NLST dataset to screen and annotate the specific locations of pulmonary nodules and created a new dataset named NLST-cmst. Our experimental results on the NLST-cmst dataset demonstrate significant performance improvements, with an accuracy of 0.8974, a precision of 0.8235, an F1 score of 0.8750, an AUC of 0.9389, and a recall of 0.9333. These findings indicate that our multimodal spatiotemporal fusion approach, which combines follow-up data with clinical information, surpasses existing methods, underscoring its effectiveness in predicting nodule malignancy.