Learning to Associate Every Segment for Video Panoptic Segmentation

TL;DR

This paper introduces a novel approach for video panoptic segmentation that learns segment and pixel-level temporal correspondence simultaneously, achieving state-of-the-art accuracy with high efficiency and no additional inference cost.

Contribution

The paper proposes two new learning objectives for coarse segment-level and fine pixel-level matching, enabling efficient, per-frame inference for video panoptic segmentation.

Findings

Achieves state-of-the-art results on Cityscapes-VPS and VIPER datasets.

Runs three times faster than previous methods.

No extra computation during inference.

Abstract

Temporal correspondence - linking pixels or objects across frames - is a fundamental supervisory signal for the video models. For the panoptic understanding of dynamic scenes, we further extend this concept to every segment. Specifically, we aim to learn coarse segment-level matching and fine pixel-level matching together. We implement this idea by designing two novel learning objectives. To validate our proposals, we adopt a deep siamese model and train the model to learn the temporal correspondence on two different levels (i.e., segment and pixel) along with the target task. At inference time, the model processes each frame independently without any extra computation and post-processing. We show that our per-frame inference model can achieve new state-of-the-art results on Cityscapes-VPS and VIPER datasets. Moreover, due to its high efficiency, the model runs in a fraction of time (3x) compared to the previous state-of-the-art approach.