Multimodal Survival Analysis with Locally Deployable Large Language Models

TL;DR

This paper presents a multimodal survival analysis method that combines clinical text, tabular data, and genomic profiles using lightweight, locally deployable large language models, ensuring privacy, calibration, and improved prognostic accuracy.

Contribution

It introduces a novel approach that integrates multimodal data with locally deployable LLMs, employing teacher-student distillation for calibrated survival predictions and evidence-grounded prognosis generation.

Findings

Outperforms standard baselines on TCGA cohort

Ensures privacy by avoiding cloud reliance

Reduces hallucination and miscalibration risks

Abstract

We study multimodal survival analysis integrating clinical text, tabular covariates, and genomic profiles using locally deployable large language models (LLMs). As many institutions face tight computational and privacy constraints, this setting motivates the use of lightweight, on-premises models. Our approach jointly estimates calibrated survival probabilities and generates concise, evidence-grounded prognosis text via teacher-student distillation and principled multimodal fusion. On a TCGA cohort, it outperforms standard baselines, avoids reliance on cloud services and associated privacy concerns, and reduces the risk of hallucinated or miscalibrated estimates that can be observed in base LLMs.