LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction

TL;DR

This paper introduces KEMM, a novel multimodal cancer survival prediction model that leverages expert reports and background knowledge generated by large language models to improve feature discrimination and prediction accuracy.

Contribution

The paper presents a new LLM-driven approach that integrates expert reports and background knowledge into multimodal survival prediction, enhancing feature extraction and model performance.

Findings

Achieves state-of-the-art results on five datasets.

Effectively leverages LLM-generated knowledge for survival prediction.

Improves feature discrimination from high-dimensional modalities.

Abstract

Current multimodal survival prediction methods typically rely on pathology images (WSIs) and genomic data, both of which are high-dimensional and redundant, making it difficult to extract discriminative features from them and align different modalities. Moreover, using a simple survival follow-up label is insufficient to supervise such a complex task. To address these challenges, we propose KEMM, an LLM-driven Knowledge-Enhanced Multimodal Model for cancer survival prediction, which integrates expert reports and prognostic background knowledge. 1) Expert reports, provided by pathologists on a case-by-case basis and refined by large language model (LLM), offer succinct and clinically focused diagnostic statements. This information may typically suggest different survival outcomes. 2) Prognostic background knowledge (PBK), generated concisely by LLM, provides valuable prognostic background knowledge on different cancer types, which also enhances survival prediction. To leverage these knowledge, we introduce the knowledge-enhanced cross-modal (KECM) attention module. KECM can effectively guide the network to focus on discriminative and survival-relevant features from highly redundant modalities. Extensive experiments on five datasets demonstrate that KEMM achieves state-of-the-art performance. The code will be released upon acceptance.