TMI-CLNet: Triple-Modal Interaction Network for Chronic Liver Disease Prognosis From Imaging, Clinical, and Radiomic Data Fusion

TL;DR

TMI-CLNet is a novel neural network that effectively fuses imaging, radiomic, and clinical data for improved prognosis of chronic liver disease by addressing heterogeneity and inter-modal relationships.

Contribution

The paper introduces a triple-modal interaction network with specialized modules and a fusion loss to enhance multimodal data integration in medical prognosis.

Findings

Outperforms existing unimodal and multimodal methods on liver prognosis data

Effectively captures inter-modal relationships and reduces intra-modality redundancy

Demonstrates significant improvement in prognostic accuracy

Abstract

Chronic liver disease represents a significant health challenge worldwide and accurate prognostic evaluations are essential for personalized treatment plans. Recent evidence suggests that integrating multimodal data, such as computed tomography imaging, radiomic features, and clinical information, can provide more comprehensive prognostic information. However, modalities have an inherent heterogeneity, and incorporating additional modalities may exacerbate the challenges of heterogeneous data fusion. Moreover, existing multimodal fusion methods often struggle to adapt to richer medical modalities, making it difficult to capture inter-modal relationships. To overcome these limitations, We present the Triple-Modal Interaction Chronic Liver Network (TMI-CLNet). Specifically, we develop an Intra-Modality Aggregation module and a Triple-Modal Cross-Attention Fusion module, which are designed to eliminate intra-modality redundancy and extract cross-modal information, respectively. Furthermore, we design a Triple-Modal Feature Fusion loss function to align feature representations across modalities. Extensive experiments on the liver prognosis dataset demonstrate that our approach significantly outperforms existing state-of-the-art unimodal models and other multi-modal techniques. Our code is available at https://github.com/Mysterwll/liver.git.