Exploring Zero-Shot Anomaly Detection with CLIP in Medical Imaging: Are We There Yet?

TL;DR

This paper evaluates the effectiveness of CLIP-based models for zero-shot anomaly detection in medical imaging, specifically brain tumor detection, highlighting their potential and current limitations for clinical application.

Contribution

It provides the first assessment of CLIP models' ability to detect medical anomalies without task-specific training, revealing both promise and challenges.

Findings

CLIP models transfer general knowledge to medical tasks.

Performance is currently insufficient for clinical use.

Further adaptation is necessary for reliable medical anomaly detection.

Abstract

Zero-shot anomaly detection (ZSAD) offers potential for identifying anomalies in medical imaging without task-specific training. In this paper, we evaluate CLIP-based models, originally developed for industrial tasks, on brain tumor detection using the BraTS-MET dataset. Our analysis examines their ability to detect medical-specific anomalies with no or minimal supervision, addressing the challenges posed by limited data annotation. While these models show promise in transferring general knowledge to medical tasks, their performance falls short of the precision required for clinical use. Our findings highlight the need for further adaptation before CLIP-based models can be reliably applied to medical anomaly detection.