Improving Online Performance Prediction for Semantic Segmentation

TL;DR

This paper introduces an improved online performance prediction method for semantic segmentation DNNs, utilizing auxiliary monocular depth estimation and multi-task learning to enhance accuracy in safety-critical applications.

Contribution

It proposes a novel multi-task training approach with shared encoder-decoder architecture and temporal statistics aggregation for better online performance prediction.

Findings

Enhanced prediction accuracy on KITTI dataset

Shared encoder improves computational efficiency

Temporal aggregation reduces prediction error

Abstract

In this work we address the task of observing the performance of a semantic segmentation deep neural network (DNN) during online operation, i.e., during inference, which is of high importance in safety-critical applications such as autonomous driving. Here, many high-level decisions rely on such DNNs, which are usually evaluated offline, while their performance in online operation remains unknown. To solve this problem, we propose an improved online performance prediction scheme, building on a recently proposed concept of predicting the primary semantic segmentation task's performance. This can be achieved by evaluating the auxiliary task of monocular depth estimation with a measurement supplied by a LiDAR sensor and a subsequent regression to the semantic segmentation performance. In particular, we propose (i) sequential training methods for both tasks in a multi-task training setup, (ii) to share the encoder as well as parts of the decoder between both task's networks for improved efficiency, and (iii) a temporal statistics aggregation method, which significantly reduces the performance prediction error at the cost of a small algorithmic latency. Evaluation on the KITTI dataset shows that all three aspects improve the performance prediction compared to previous approaches.