Uncertainty-aware Panoptic Segmentation

TL;DR

This paper introduces uncertainty-aware panoptic segmentation, combining semantic and instance segmentation with uncertainty estimates, and proposes new metrics and a novel network architecture to improve safety and reliability in scene understanding.

Contribution

It presents the first framework for uncertainty-aware panoptic segmentation, including new metrics and the EvPSNet architecture that leverages uncertainty for improved performance.

Findings

EvPSNet achieves state-of-the-art PQ scores

Proposes new metrics uPQ and pECE for uncertainty evaluation

Demonstrates the importance of uncertainty in scene understanding

Abstract

Reliable scene understanding is indispensable for modern autonomous systems. Current learning-based methods typically try to maximize their performance based on segmentation metrics that only consider the quality of the segmentation. However, for the safe operation of a system in the real world it is crucial to consider the uncertainty in the prediction as well. In this work, we introduce the novel task of uncertainty-aware panoptic segmentation, which aims to predict per-pixel semantic and instance segmentations, together with per-pixel uncertainty estimates. We define two novel metrics to facilitate its quantitative analysis, the uncertainty-aware Panoptic Quality (uPQ) and the panoptic Expected Calibration Error (pECE). We further propose the novel top-down Evidential Panoptic Segmentation Network (EvPSNet) to solve this task. Our architecture employs a simple yet effective panoptic fusion module that leverages the predicted uncertainties. Furthermore, we provide several strong baselines combining state-of-the-art panoptic segmentation networks with sampling-free uncertainty estimation techniques. Extensive evaluations show that our EvPSNet achieves the new state-of-the-art for the standard Panoptic Quality (PQ), as well as for our uncertainty-aware panoptic metrics. We make the code available at: \url{https://github.com/kshitij3112/EvPSNet}