PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection

TL;DR

PointAD leverages CLIP's generalization to detect 3D anomalies in unseen objects by integrating point cloud and pixel data through a unified 3D-2D rendering and hybrid representation learning framework.

Contribution

This work introduces PointAD, a novel zero-shot 3D anomaly detection method that combines 3D and 2D information using hybrid learning and rendering techniques, enabling detection without prior training samples.

Findings

Outperforms existing methods in zero-shot 3D anomaly detection tasks.

Effectively integrates RGB, 3D point clouds, and 2D renderings for improved anomaly understanding.

Demonstrates strong generalization across diverse unseen objects.

Abstract

Zero-shot (ZS) 3D anomaly detection is a crucial yet unexplored field that addresses scenarios where target 3D training samples are unavailable due to practical concerns like privacy protection. This paper introduces PointAD, a novel approach that transfers the strong generalization capabilities of CLIP for recognizing 3D anomalies on unseen objects. PointAD provides a unified framework to comprehend 3D anomalies from both points and pixels. In this framework, PointAD renders 3D anomalies into multiple 2D renderings and projects them back into 3D space. To capture the generic anomaly semantics into PointAD, we propose hybrid representation learning that optimizes the learnable text prompts from 3D and 2D through auxiliary point clouds. The collaboration optimization between point and pixel representations jointly facilitates our model to grasp underlying 3D anomaly patterns, contributing to detecting and segmenting anomalies of unseen diverse 3D objects. Through the alignment of 3D and 2D space, our model can directly integrate RGB information, further enhancing the understanding of 3D anomalies in a plug-and-play manner. Extensive experiments show the superiority of PointAD in ZS 3D anomaly detection across diverse unseen objects.