Investigating the Semantic Robustness of CLIP-based Zero-Shot Anomaly Segmentation

TL;DR

This paper evaluates the robustness of CLIP-based zero-shot anomaly segmentation under semantic perturbations, revealing significant performance drops across different models and highlighting the need for careful robustness assessment.

Contribution

It systematically investigates the semantic robustness of WinCLIP zero-shot anomaly segmentation under various perturbations, providing empirical performance bounds and insights.

Findings

Performance drops up to 20% in ROC AUC

Performance drops up to 40% in per-region overlap

Robustness issues are consistent across different CLIP backbones

Abstract

Zero-shot anomaly segmentation using pre-trained foundation models is a promising approach that enables effective algorithms without expensive, domain-specific training or fine-tuning. Ensuring that these methods work across various environmental conditions and are robust to distribution shifts is an open problem. We investigate the performance of WinCLIP [14] zero-shot anomaly segmentation algorithm by perturbing test data using three semantic transformations: bounded angular rotations, bounded saturation shifts, and hue shifts. We empirically measure a lower performance bound by aggregating across per-sample worst-case perturbations and find that average performance drops by up to 20% in area under the ROC curve and 40% in area under the per-region overlap curve. We find that performance is consistently lowered on three CLIP backbones, regardless of model architecture or learning objective, demonstrating a need for careful performance evaluation.