PanopticDepth: A Unified Framework for Depth-aware Panoptic Segmentation

TL;DR

This paper introduces PanopticDepth, a unified framework that jointly performs depth-aware panoptic segmentation by using instance-specific kernels and depth cues, improving 3D scene reconstruction from a single image.

Contribution

It proposes a novel unified approach applying dynamic convolution for joint depth and segmentation prediction, exploiting mutual benefits between tasks for better accuracy.

Findings

Effective on Cityscapes-DPS and SemKITTI-DPS datasets.

Outperforms previous methods in depth accuracy and segmentation quality.

Provides a new paradigm for depth-aware panoptic segmentation.

Abstract

This paper presents a unified framework for depth-aware panoptic segmentation (DPS), which aims to reconstruct 3D scene with instance-level semantics from one single image. Prior works address this problem by simply adding a dense depth regression head to panoptic segmentation (PS) networks, resulting in two independent task branches. This neglects the mutually-beneficial relations between these two tasks, thus failing to exploit handy instance-level semantic cues to boost depth accuracy while also producing sub-optimal depth maps. To overcome these limitations, we propose a unified framework for the DPS task by applying a dynamic convolution technique to both the PS and depth prediction tasks. Specifically, instead of predicting depth for all pixels at a time, we generate instance-specific kernels to predict depth and segmentation masks for each instance. Moreover, leveraging the instance-wise depth estimation scheme, we add additional instance-level depth cues to assist with supervising the depth learning via a new depth loss. Extensive experiments on Cityscapes-DPS and SemKITTI-DPS show the effectiveness and promise of our method. We hope our unified solution to DPS can lead a new paradigm in this area. Code is available at https://github.com/NaiyuGao/PanopticDepth.