GOSS: Towards Generalized Open-set Semantic Segmentation

TL;DR

This paper introduces GOSS, a new image segmentation task that combines open-set and generic segmentation to better understand both known and unknown regions, supported by a new metric, benchmarks, and a baseline neural model.

Contribution

The paper proposes GOSS, unifying open-set and generic segmentation, along with a new evaluation metric, benchmark datasets, and a baseline neural architecture for the task.

Findings

The baseline model effectively performs GOSS on multiple benchmarks.

The new metric balances pixel classification and clustering.

GOSS enhances image understanding by jointly handling known and unknown regions.

Abstract

In this paper, we present and study a new image segmentation task, called Generalized Open-set Semantic Segmentation (GOSS). Previously, with the well-known open-set semantic segmentation (OSS), the intelligent agent only detects the unknown regions without further processing, limiting their perception of the environment. It stands to reason that a further analysis of the detected unknown pixels would be beneficial. Therefore, we propose GOSS, which unifies the abilities of two well-defined segmentation tasks, OSS and generic segmentation (GS), in a holistic way. Specifically, GOSS classifies pixels as belonging to known classes, and clusters (or groups) of pixels of unknown class are labelled as such. To evaluate this new expanded task, we further propose a metric which balances the pixel classification and clustering aspects. Moreover, we build benchmark tests on top of existing datasets and propose a simple neural architecture as a baseline, which jointly predicts pixel classification and clustering under open-set settings. Our experiments on multiple benchmarks demonstrate the effectiveness of our baseline. We believe our new GOSS task can produce an expressive image understanding for future research. Code will be made available.