Self-Supervised Anomaly Detection in the Wild: Favor Joint Embeddings Methods

TL;DR

This study evaluates various self-supervised learning methods for real-world anomaly detection in sewer infrastructure, demonstrating that joint-embedding approaches outperform reconstruction-based methods, and emphasizing the importance of SSL model selection and evaluation.

Contribution

It provides a comprehensive evaluation of SSL methods for anomaly detection in infrastructure, highlighting the superiority of joint-embedding techniques and the need for better label-free assessment metrics.

Findings

Joint-embedding methods like SimCLR and Barlow Twins outperform reconstruction-based approaches.

SSL model choice impacts performance more than backbone architecture.

Current evaluation metrics like RankMe are inadequate for assessing SSL representations.

Abstract

Accurate anomaly detection is critical in vision-based infrastructure inspection, where it helps prevent costly failures and enhances safety. Self-Supervised Learning (SSL) offers a promising approach by learning robust representations from unlabeled data. However, its application in anomaly detection remains underexplored. This paper addresses this gap by providing a comprehensive evaluation of SSL methods for real-world anomaly detection, focusing on sewer infrastructure. Using the Sewer-ML dataset, we evaluate lightweight models such as ViT-Tiny and ResNet-18 across SSL frameworks, including BYOL, Barlow Twins, SimCLR, DINO, and MAE, under varying class imbalance levels. Through 250 experiments, we rigorously assess the performance of these SSL methods to ensure a robust and comprehensive evaluation. Our findings highlight the superiority of joint-embedding methods like SimCLR and Barlow Twins over reconstruction-based approaches such as MAE, which struggle to maintain performance under class imbalance. Furthermore, we find that the SSL model choice is more critical than the backbone architecture. Additionally, we emphasize the need for better label-free assessments of SSL representations, as current methods like RankMe fail to adequately evaluate representation quality, making cross-validation without labels infeasible. Despite the remaining performance gap between SSL and supervised models, these findings highlight the potential of SSL to enhance anomaly detection, paving the way for further research in this underexplored area of SSL applications.