Label-Efficient Chest X-ray Diagnosis via Partial CLIP Adaptation

TL;DR

This paper proposes a label-efficient method for chest X-ray diagnosis using partial CLIP adaptation, demonstrating significant improvements in disease classification with limited labeled data, suitable for real-world hospital scenarios.

Contribution

It introduces a partial fine-tuning approach of CLIP's visual encoder for few-shot chest X-ray diagnosis, reflecting practical hospital workflows with sparse annotations.

Findings

Over 20% improvement in mean AUC with few-shot learning

Effective adaptation of CLIP for medical imaging tasks

Suitable for diagnosing both common and rare diseases

Abstract

Modern deep learning implementations for medical imaging usually rely on large labeled datasets. These datasets are often difficult to obtain due to privacy concerns, high costs, and even scarcity of cases. In this paper, a label-efficient strategy is proposed for chest X-ray diagnosis that seeks to reflect real-world hospital scenarios. The experiments use the NIH Chest X-ray14 dataset and a pre-trained CLIP ViT-B/32 model. The model is adapted via partial fine-tuning of its visual encoder and then evaluated using zero-shot and few-shot learning with 1-16 labeled examples per disease class. The tests demonstrate that CLIP's pre-trained vision-language features can be effectively adapted to few-shot medical imaging tasks, achieving over 20\% improvement in mean AUC score as compared to the zero-shot baseline. The key aspect of this work is to attempt to simulate internal hospital workflows, where image archives exist but annotations are sparse. This work evaluates a practical and scalable solution for both common and rare disease diagnosis. Additionally this research is intended for academic and experimental purposes only and has not been peer reviewed yet. All code is found at https://github.com/heet007-code/CLIP-disease-xray.