Exploring the Versatility of Zero-Shot CLIP for Interstitial Lung Disease Classification

TL;DR

This paper demonstrates that zero-shot CLIP, combined with domain adaptive pretraining, can effectively classify interstitial lung diseases from CT scans without labeled data, showcasing its versatility in medical imaging.

Contribution

The study introduces a novel workflow integrating zero-shot CLIP and DAPT for ILD classification, highlighting its effectiveness without requiring labeled training data.

Findings

Achieved an AUROC of 0.893 in ILD classification

Demonstrated strong zero-shot performance with CLIP and DAPT

Showcased potential for label-efficient medical imaging diagnosis

Abstract

Interstitial lung diseases (ILD) present diagnostic challenges due to their varied manifestations and overlapping imaging features. To address this, we propose a machine learning approach that utilizes CLIP, a multimodal (image and text) self-supervised model, for ILD classification. We extensively integrate zero-shot CLIP throughout our workflow, starting from the initial extraction of image patches from volumetric CT scans and proceeding to ILD classification using "patch montages". Furthermore, we investigate how domain adaptive pretraining (DAPT) CLIP with task-specific images (CT "patch montages" extracted with ILD-specific prompts for CLIP) and/or text (lung-specific sections of radiology reports) affects downstream ILD classification performance. By leveraging CLIP-extracted "patch montages" and DAPT, we achieve strong zero-shot ILD classification results, including an AUROC of 0.893, without the need for any labeled training data. This work highlights the versatility and potential of multimodal models like CLIP for medical image classification tasks where labeled data is scarce.