Far Away in the Deep Space: Dense Nearest-Neighbor-Based Out-of-Distribution Detection

TL;DR

This paper demonstrates that nearest-neighbor methods, especially with transformer-based features, achieve state-of-the-art out-of-distribution detection in complex driving scenes without retraining or affecting primary tasks.

Contribution

It shows that dense nearest-neighbor approaches with transformer features outperform existing methods in out-of-distribution detection for complex scenes, and transfers some benefits to CNNs.

Findings

Transformer features improve similarity metrics for OOD detection.

Nearest-neighbor approach achieves state-of-the-art results on multiple benchmarks.

The method does not impact primary segmentation performance.

Abstract

The key to out-of-distribution detection is density estimation of the in-distribution data or of its feature representations. This is particularly challenging for dense anomaly detection in domains where the in-distribution data has a complex underlying structure. Nearest-Neighbors approaches have been shown to work well in object-centric data domains, such as industrial inspection and image classification. In this paper, we show that nearest-neighbor approaches also yield state-of-the-art results on dense novelty detection in complex driving scenes when working with an appropriate feature representation. In particular, we find that transformer-based architectures produce representations that yield much better similarity metrics for the task. We identify the multi-head structure of these models as one of the reasons, and demonstrate a way to transfer some of the improvements to CNNs. Ultimately, the approach is simple and non-invasive, i.e., it does not affect the primary segmentation performance, refrains from training on examples of anomalies, and achieves state-of-the-art results on RoadAnomaly, StreetHazards, and SegmentMeIfYouCan-Anomaly.