UltraAD: Fine-Grained Ultrasound Anomaly Classification via Few-Shot CLIP Adaptation

TL;DR

UltraAD introduces a vision-language model that leverages few-shot ultrasound data and text prompts for precise anomaly localization and fine-grained classification, addressing domain gaps and differentiation challenges in medical ultrasound imaging.

Contribution

The paper presents UltraAD, a novel VLM-based approach that enhances ultrasound anomaly detection and classification using few-shot learning and image-informed prompts, improving localization and differentiation accuracy.

Findings

Outperforms state-of-the-art in lesion localization

Achieves superior fine-grained classification accuracy

Effective across multiple breast US datasets

Abstract

Precise anomaly detection in medical images is critical for clinical decision-making. While recent unsupervised or semi-supervised anomaly detection methods trained on large-scale normal data show promising results, they lack fine-grained differentiation, such as benign vs. malignant tumors. Additionally, ultrasound (US) imaging is highly sensitive to devices and acquisition parameter variations, creating significant domain gaps in the resulting US images. To address these challenges, we propose UltraAD, a vision-language model (VLM)-based approach that leverages few-shot US examples for generalized anomaly localization and fine-grained classification. To enhance localization performance, the image-level token of query visual prototypes is first fused with learnable text embeddings. This image-informed prompt feature is then further integrated with patch-level tokens, refining local representations for improved accuracy. For fine-grained classification, a memory bank is constructed from few-shot image samples and corresponding text descriptions that capture anatomical and abnormality-specific features. During training, the stored text embeddings remain frozen, while image features are adapted to better align with medical data. UltraAD has been extensively evaluated on three breast US datasets, outperforming state-of-the-art methods in both lesion localization and fine-grained medical classification. The code will be released upon acceptance.