MADPOT: Medical Anomaly Detection with CLIP Adaptation and Partial Optimal Transport

TL;DR

MADPOT introduces a novel medical anomaly detection method that adapts CLIP using visual prompts, partial optimal transport, and contrastive learning, achieving state-of-the-art results across various scenarios without synthetic data.

Contribution

The paper presents a new approach combining visual adapters, prompt learning, and partial optimal transport to enhance CLIP's performance in medical anomaly detection.

Findings

Achieves state-of-the-art results in few-shot, zero-shot, and cross-dataset scenarios.

Effectively captures subtle abnormalities using multiple prompts aligned with local features.

Does not rely on synthetic data or memory banks.

Abstract

Medical anomaly detection (AD) is challenging due to diverse imaging modalities, anatomical variations, and limited labeled data. We propose a novel approach combining visual adapters and prompt learning with Partial Optimal Transport (POT) and contrastive learning (CL) to improve CLIP's adaptability to medical images, particularly for AD. Unlike standard prompt learning, which often yields a single representation, our method employs multiple prompts aligned with local features via POT to capture subtle abnormalities. CL further enforces intra-class cohesion and inter-class separation. Our method achieves state-of-the-art results in few-shot, zero-shot, and cross-dataset scenarios without synthetic data or memory banks. The code is available at https://github.com/mahshid1998/MADPOT.