Open-World Semantic Segmentation Including Class Similarity

TL;DR

This paper introduces a novel open-world semantic segmentation method that accurately classifies known objects, detects unseen classes, and measures their similarity to known categories, enhancing autonomous systems' understanding of complex environments.

Contribution

It presents a new approach that combines closed-world segmentation with unknown class detection and similarity measurement without additional training data.

Findings

Achieves state-of-the-art results on known and unknown classes

Effectively detects and distinguishes between multiple unseen classes

Provides meaningful similarity scores for new categories

Abstract

Interpreting camera data is key for autonomously acting systems, such as autonomous vehicles. Vision systems that operate in real-world environments must be able to understand their surroundings and need the ability to deal with novel situations. This paper tackles open-world semantic segmentation, i.e., the variant of interpreting image data in which objects occur that have not been seen during training. We propose a novel approach that performs accurate closed-world semantic segmentation and, at the same time, can identify new categories without requiring any additional training data. Our approach additionally provides a similarity measure for every newly discovered class in an image to a known category, which can be useful information in downstream tasks such as planning or mapping. Through extensive experiments, we show that our model achieves state-of-the-art results on classes known from training data as well as for anomaly segmentation and can distinguish between different unknown classes.