Consensus Synergizes with Memory: A Simple Approach for Anomaly Segmentation in Urban Scenes

TL;DR

This paper introduces CosMe, a simple yet effective method that combines a memory bank of seen prototypes with a consensus mechanism to improve anomaly segmentation in urban scenes, especially for detecting out-of-distribution objects.

Contribution

The paper proposes CosMe, a novel approach that leverages a memory bank and consensus of features to better distinguish in-distribution and OOD samples in urban scene anomaly segmentation.

Findings

CosMe outperforms previous methods on urban scene datasets.

The consensus mechanism effectively distinguishes hard in-distribution from OOD samples.

Memory bank of prototypes enhances anomaly detection accuracy.

Abstract

Anomaly segmentation is a crucial task for safety-critical applications, such as autonomous driving in urban scenes, where the goal is to detect out-of-distribution (OOD) objects with categories which are unseen during training. The core challenge of this task is how to distinguish hard in-distribution samples from OOD samples, which has not been explicitly discussed yet. In this paper, we propose a novel and simple approach named Consensus Synergizes with Memory (CosMe) to address this challenge, inspired by the psychology finding that groups perform better than individuals on memory tasks. The main idea is 1) building a memory bank which consists of seen prototypes extracted from multiple layers of the pre-trained segmentation model and 2) training an auxiliary model that mimics the behavior of the pre-trained model, and then measuring the consensus of their mid-level features as complementary cues that synergize with the memory bank. CosMe is good at distinguishing between hard in-distribution examples and OOD samples. Experimental results on several urban scene anomaly segmentation datasets show that CosMe outperforms previous approaches by large margins.