Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentation

TL;DR

Video-kMaX introduces a unified framework for online and near-online video panoptic segmentation, enabling adaptable, state-of-the-art scene understanding across various video analysis scenarios.

Contribution

The paper presents a novel unified architecture with clip-kMaX and HiLA-MB components, bridging the gap between online and near-online VPS methods.

Findings

Achieves new state-of-the-art on KITTI-STEP and VIPSeg datasets.

Outperforms previous methods on VSPW for video semantic segmentation.

Unifies online and near-online VPS with a flexible, general approach.

Abstract

Video Panoptic Segmentation (VPS) aims to achieve comprehensive pixel-level scene understanding by segmenting all pixels and associating objects in a video. Current solutions can be categorized into online and near-online approaches. Evolving over the time, each category has its own specialized designs, making it nontrivial to adapt models between different categories. To alleviate the discrepancy, in this work, we propose a unified approach for online and near-online VPS. The meta architecture of the proposed Video-kMaX consists of two components: within clip segmenter (for clip-level segmentation) and cross-clip associater (for association beyond clips). We propose clip-kMaX (clip k-means mask transformer) and HiLA-MB (Hierarchical Location-Aware Memory Buffer) to instantiate the segmenter and associater, respectively. Our general formulation includes the online scenario as a special case by adopting clip length of one. Without bells and whistles, Video-kMaX sets a new state-of-the-art on KITTI-STEP and VIPSeg for video panoptic segmentation, and VSPW for video semantic segmentation. Code will be made publicly available.