ProPanDL: A Modular Architecture for Uncertainty-Aware Panoptic Segmentation

TL;DR

ProPanDL introduces a modular neural network architecture for uncertainty-aware panoptic segmentation, capable of estimating full probability distributions for semantic and spatial aspects, with new evaluation metrics for uncertainty.

Contribution

It is the first to provide a unified framework for uncertainty estimation in panoptic segmentation with novel metrics and robust evaluation methods.

Findings

ProPanDL achieves well-calibrated uncertainty estimates.

The approach maintains high segmentation performance.

New metrics effectively evaluate spatial and semantic uncertainty.

Abstract

We introduce ProPanDL, a family of networks capable of uncertainty-aware panoptic segmentation. Unlike existing segmentation methods, ProPanDL is capable of estimating full probability distributions for both the semantic and spatial aspects of panoptic segmentation. We implement and evaluate ProPanDL variants capable of estimating both parametric (Variance Network) and parameter-free (SampleNet) distributions quantifying pixel-wise spatial uncertainty. We couple these approaches with two methods (Temperature Scaling and Evidential Deep Learning) for semantic uncertainty estimation. To evaluate the uncertainty-aware panoptic segmentation task, we address limitations with existing approaches by proposing new metrics that enable separate evaluation of spatial and semantic uncertainty. We additionally propose the use of the energy score, a proper scoring rule, for more robust evaluation of spatial output distributions. Using these metrics, we conduct an extensive evaluation of ProPanDL variants. Our results demonstrate that ProPanDL is capable of estimating well-calibrated and meaningful output distributions while still retaining strong performance on the base panoptic segmentation task.