1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation

TL;DR

This paper presents a top-performing approach for video panoptic and semantic segmentation in challenging wild scenes, achieving state-of-the-art results in the PVUW 2024 challenge by leveraging vision transformers and multi-stage segmentation frameworks.

Contribution

The paper introduces a novel combination of vision transformer models and multi-stage segmentation frameworks that achieves first place in video panoptic segmentation and high ranking in semantic segmentation challenges.

Findings

Achieved first place in PVUW 2024 VPS challenge with top metrics.

Secured third place in VSS challenge with minor fine-tuning.

Established state-of-the-art results on challenging wild video datasets.

Abstract

The third Pixel-level Video Understanding in the Wild (PVUW CVPR 2024) challenge aims to advance the state of art in video understanding through benchmarking Video Panoptic Segmentation (VPS) and Video Semantic Segmentation (VSS) on challenging videos and scenes introduced in the large-scale Video Panoptic Segmentation in the Wild (VIPSeg) test set and the large-scale Video Scene Parsing in the Wild (VSPW) test set, respectively. This paper details our research work that achieved the 1st place winner in the PVUW'24 VPS challenge, establishing state of art results in all metrics, including the Video Panoptic Quality (VPQ) and Segmentation and Tracking Quality (STQ). With minor fine-tuning our approach also achieved the 3rd place in the PVUW'24 VSS challenge ranked by the mIoU (mean intersection over union) metric and the first place ranked by the VC16 (16-frame video consistency) metric. Our winning solution stands on the shoulders of giant foundational vision transformer model (DINOv2 ViT-g) and proven multi-stage Decoupled Video Instance Segmentation (DVIS) frameworks for video understanding.