A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer

TL;DR

This study introduces RenalCLIP, a disease-specific vision-language model trained on extensive CT data, which improves diagnosis, prognosis, and management of kidney cancer with high accuracy and data efficiency.

Contribution

The paper presents RenalCLIP, a novel disease-centric foundation model for renal mass analysis, demonstrating superior performance and generalizability over existing models in multiple clinical tasks.

Findings

RenalCLIP outperforms state-of-the-art models in 10 clinical tasks.

Achieves a 20% higher C-index in recurrence-free survival prediction.

Requires only 20% of training data to reach peak diagnostic performance.

Abstract

The non-invasive assessment of increasingly incidentally discovered renal masses is a critical challenge in urologic oncology, where diagnostic uncertainty frequently leads to the overtreatment of benign or indolent tumors. In this study, we developed and validated RenalCLIP using a dataset of 27,866 CT scans from 8,809 patients across nine Chinese medical centers and the public TCIA cohort, a visual-language foundation model for characterization, diagnosis and prognosis of renal mass. The model was developed via a two-stage pre-training strategy that first enhances the image and text encoders with domain-specific knowledge before aligning them through a contrastive learning objective, to create robust representations for superior generalization and diagnostic precision. RenalCLIP achieved better performance and superior generalizability across 10 core tasks spanning the full clinical workflow of kidney cancer, including anatomical assessment, diagnostic classification, and survival prediction, compared with other state-of-the-art general-purpose CT foundation models. Especially, for complicated task like recurrence-free survival prediction in the TCIA cohort, RenalCLIP achieved a C-index of 0.726, representing a substantial improvement of approximately 20% over the leading baselines. Furthermore, RenalCLIP's pre-training imparted remarkable data efficiency; in the diagnostic classification task, it only needs 20% training data to achieve the peak performance of all baseline models even after they were fully fine-tuned on 100% of the data. Additionally, it achieved superior performance in report generation, image-text retrieval and zero-shot diagnosis tasks. Our findings establish that RenalCLIP provides a robust tool with the potential to enhance diagnostic accuracy, refine prognostic stratification, and personalize the management of patients with kidney cancer.