Balancing Shared and Task-Specific Representations: A Hybrid Approach to Depth-Aware Video Panoptic Segmentation

TL;DR

This paper introduces Multiformer, a hybrid transformer-based model for depth-aware video panoptic segmentation that effectively balances shared and task-specific representations, achieving state-of-the-art results.

Contribution

It proposes a hybrid multi-task decoder with task-specific branches and shared representations, advancing depth-aware video segmentation techniques.

Findings

Outperforms previous methods by 3.0 DVPQ points with ResNet-50 backbone.

Further improves by 4.0 DVPQ points with Swin-B backbone.

Achieves state-of-the-art performance on Cityscapes-DVPS and SemKITTI-DVPS datasets.

Abstract

In this work, we present Multiformer, a novel approach to depth-aware video panoptic segmentation (DVPS) based on the mask transformer paradigm. Our method learns object representations that are shared across segmentation, monocular depth estimation, and object tracking subtasks. In contrast to recent unified approaches that progressively refine a common object representation, we propose a hybrid method using task-specific branches within each decoder block, ultimately fusing them into a shared representation at the block interfaces. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that Multiformer achieves state-of-the-art performance across all DVPS metrics, outperforming previous methods by substantial margins. With a ResNet-50 backbone, Multiformer surpasses the previous best result by 3.0 DVPQ points while also improving depth estimation accuracy. Using a Swin-B backbone, Multiformer further improves performance by 4.0 DVPQ points. Multiformer also provides valuable insights into the design of multi-task decoder architectures.