Uncertainty and Prediction Quality Estimation for Semantic Segmentation via Graph Neural Networks

TL;DR

This paper introduces a novel approach using graph neural networks to improve uncertainty and prediction quality estimation for semantic segmentation by leveraging relationships between neighboring segments.

Contribution

It proposes a segment-wise uncertainty estimation method that incorporates neighboring segment information through GNNs, enhancing prediction quality assessment.

Findings

GNNs outperform previous methods in uncertainty estimation.

Modeling segment relationships improves prediction quality accuracy.

Different GNN architectures show varying levels of performance.

Abstract

When employing deep neural networks (DNNs) for semantic segmentation in safety-critical applications like automotive perception or medical imaging, it is important to estimate their performance at runtime, e.g. via uncertainty estimates or prediction quality estimates. Previous works mostly performed uncertainty estimation on pixel-level. In a line of research, a connected-component-wise (segment-wise) perspective was taken, approaching uncertainty estimation on an object-level by performing so-called meta classification and regression to estimate uncertainty and prediction quality, respectively. In those works, each predicted segment is considered individually to estimate its uncertainty or prediction quality. However, the neighboring segments may provide additional hints on whether a given predicted segment is of high quality, which we study in the present work. On the basis of uncertainty indicating metrics on segment-level, we use graph neural networks (GNNs) to model the relationship of a given segment's quality as a function of the given segment's metrics as well as those of its neighboring segments. We compare different GNN architectures and achieve a notable performance improvement.