Region-Aware Metric Learning for Open World Semantic Segmentation via Meta-Channel Aggregation

TL;DR

This paper introduces region-aware metric learning (RAML) with a meta-channel aggregation module to enhance open-world semantic segmentation, especially for anomaly detection and incremental few-shot learning, achieving state-of-the-art results.

Contribution

The paper proposes RAML and MCA modules to improve anomaly region segmentation and out-of-distribution object detection in open-world semantic segmentation.

Findings

RAML achieves SOTA performance on Lost And Found and Road Anomaly datasets.

MCA module effectively separates anomaly sub-regions, boosting model accuracy.

Extensive experiments validate the effectiveness of the proposed methods.

Abstract

As one of the most challenging and practical segmentation tasks, open-world semantic segmentation requires the model to segment the anomaly regions in the images and incrementally learn to segment out-of-distribution (OOD) objects, especially under a few-shot condition. The current state-of-the-art (SOTA) method, Deep Metric Learning Network (DMLNet), relies on pixel-level metric learning, with which the identification of similar regions having different semantics is difficult. Therefore, we propose a method called region-aware metric learning (RAML), which first separates the regions of the images and generates region-aware features for further metric learning. RAML improves the integrity of the segmented anomaly regions. Moreover, we propose a novel meta-channel aggregation (MCA) module to further separate anomaly regions, forming high-quality sub-region candidates and thereby improving the model performance for OOD objects. To evaluate the proposed RAML, we have conducted extensive experiments and ablation studies on Lost And Found and Road Anomaly datasets for anomaly segmentation and the CityScapes dataset for incremental few-shot learning. The results show that the proposed RAML achieves SOTA performance in both stages of open world segmentation. Our code and appendix are available at https://github.com/czifan/RAML.