Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuning

TL;DR

This paper introduces a vision-language model trained on CT images and radiology reports that enhances survival prediction and generates meaningful clinical language responses.

Contribution

It presents a novel visual instruction tuning framework for 3D CT understanding that improves survival prediction and interpretability.

Findings

Outperforms baseline survival prediction methods.

Enhances prediction especially with limited clinical data.

Generates clinically meaningful language responses.

Abstract

Accurate prognostication and risk estimation are essential for guiding clinical decision-making and optimizing patient management. While radiologist-assessed features from CT scans provide valuable indicators of disease severity and outcomes, interpreting such images requires expert knowledge, and translating rich visual information into textual summaries inevitably leads to information loss. In this work, we propose a vision-language framework for 3D CT image understanding that leverages large-scale open-sourced CT images paired with radiology reports through visual instruction tuning. This pre-training enables the model to learn clinically meaningful visual-textual representations, which can then be adapted to downstream survival prediction tasks. By incorporating a survival prediction head on top of the pre-trained model, our approach improves survival prediction from CT images and clinical data while generating clinically meaningful language responses to predefined questions. Experimental results demonstrate that our method outperforms baseline methods in survival prediction, particularly, when clinical data alone is less predictive. The code will be released upon acceptance.