Depth-aware Panoptic Segmentation

TL;DR

This paper introduces a depth-aware CNN approach for panoptic segmentation that leverages 3D scene geometry to improve instance differentiation, achieving higher accuracy on Cityscapes.

Contribution

A novel depth-aware CNN architecture with a new depth-aware dice loss for improved panoptic segmentation performance.

Findings

Reduces object merging errors in panoptic segmentation.

Outperforms baseline method by 2.2% in panoptic quality.

Effectively utilizes depth information to distinguish similar objects.

Abstract

Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by 2.2% in terms of panoptic quality.