Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation

TL;DR

Mask2Anomaly introduces a mask classification approach with novel attention, contrastive learning, and refinement techniques, achieving state-of-the-art results in open-set and anomaly segmentation tasks for autonomous driving.

Contribution

It pioneers a mask-based architecture for universal open-set segmentation, integrating multiple technical innovations to improve anomaly and unknown object detection.

Findings

Achieves state-of-the-art results across multiple segmentation benchmarks.

Demonstrates effectiveness of mask classification in open-set scenarios.

Introduces novel modules for attention, contrastive learning, and false positive reduction.

Abstract

Segmenting unknown or anomalous object instances is a critical task in autonomous driving applications, and it is approached traditionally as a per-pixel classification problem. However, reasoning individually about each pixel without considering their contextual semantics results in high uncertainty around the objects' boundaries and numerous false positives. We propose a paradigm change by shifting from a per-pixel classification to a mask classification. Our mask-based method, Mask2Anomaly, demonstrates the feasibility of integrating a mask-classification architecture to jointly address anomaly segmentation, open-set semantic segmentation, and open-set panoptic segmentation. Mask2Anomaly includes several technical novelties that are designed to improve the detection of anomalies/unknown objects: i) a global masked attention module to focus individually on the foreground and background regions; ii) a mask contrastive learning that maximizes the margin between an anomaly and known classes; iii) a mask refinement solution to reduce false positives; and iv) a novel approach to mine unknown instances based on the mask-architecture properties. By comprehensive qualitative and qualitative evaluation, we show Mask2Anomaly achieves new state-of-the-art results across the benchmarks of anomaly segmentation, open-set semantic segmentation, and open-set panoptic segmentation.