Towards Deeply Unified Depth-aware Panoptic Segmentation with Bi-directional Guidance Learning

TL;DR

This paper introduces a deeply unified framework for depth-aware panoptic segmentation that jointly performs segmentation and depth estimation, leveraging bi-directional guidance learning and geometric query enhancement to improve scene understanding.

Contribution

The paper presents a novel unified approach that integrates segmentation and depth estimation with geometric query enhancement and bi-directional guidance learning, advancing the state of the art.

Findings

Achieves new state-of-the-art results on Cityscapes-DVPS and SemKITTI-DVPS datasets.

Bi-directional guidance learning improves performance even with incomplete labels.

Integrates scene geometry into object queries for better cross-task feature learning.

Abstract

Depth-aware panoptic segmentation is an emerging topic in computer vision which combines semantic and geometric understanding for more robust scene interpretation. Recent works pursue unified frameworks to tackle this challenge but mostly still treat it as two individual learning tasks, which limits their potential for exploring cross-domain information. We propose a deeply unified framework for depth-aware panoptic segmentation, which performs joint segmentation and depth estimation both in a per-segment manner with identical object queries. To narrow the gap between the two tasks, we further design a geometric query enhancement method, which is able to integrate scene geometry into object queries using latent representations. In addition, we propose a bi-directional guidance learning approach to facilitate cross-task feature learning by taking advantage of their mutual relations. Our method sets the new state of the art for depth-aware panoptic segmentation on both Cityscapes-DVPS and SemKITTI-DVPS datasets. Moreover, our guidance learning approach is shown to deliver performance improvement even under incomplete supervision labels.