CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Prediction

TL;DR

CLIP-Lung leverages textual annotations and contrastive learning to improve lung nodule malignancy prediction, enhancing accuracy and interpretability over traditional methods.

Contribution

The paper introduces a novel framework that incorporates clinical text annotations and a channel-wise prompt module into a contrastive learning setup for lung nodule classification.

Findings

Outperforms existing methods on the LIDC-IDRI dataset

Improves interpretability of model attention maps

Effectively distinguishes challenging nodule samples

Abstract

Lung nodule malignancy prediction has been enhanced by advanced deep-learning techniques and effective tricks. Nevertheless, current methods are mainly trained with cross-entropy loss using one-hot categorical labels, which results in difficulty in distinguishing those nodules with closer progression labels. Interestingly, we observe that clinical text information annotated by radiologists provides us with discriminative knowledge to identify challenging samples. Drawing on the capability of the contrastive language-image pre-training (CLIP) model to learn generalized visual representations from text annotations, in this paper, we propose CLIP-Lung, a textual knowledge-guided framework for lung nodule malignancy prediction. First, CLIP-Lung introduces both class and attribute annotations into the training of the lung nodule classifier without any additional overheads in inference. Second, we designed a channel-wise conditional prompt (CCP) module to establish consistent relationships between learnable context prompts and specific feature maps. Third, we align image features with both class and attribute features via contrastive learning, rectifying false positives and false negatives in latent space. The experimental results on the benchmark LIDC-IDRI dataset have demonstrated the superiority of CLIP-Lung, both in classification performance and interpretability of attention maps.