Static-Dynamic Class-level Perception Consistency in Video Semantic Segmentation

TL;DR

This paper introduces a novel class-level perceptual consistency framework for video semantic segmentation, leveraging class prototypes and multi-scale alignment to improve temporal consistency and segmentation accuracy.

Contribution

It proposes a static-dynamic class-level perceptual consistency framework with contrastive learning and semantic alignment modules, advancing beyond pixel-level methods.

Findings

Outperforms state-of-the-art on VSPW and Cityscapes datasets

Reduces computational cost with a window-based attention method

Enhances temporal semantic consistency in video segmentation

Abstract

Video semantic segmentation(VSS) has been widely employed in lots of fields, such as simultaneous localization and mapping, autonomous driving and surveillance. Its core challenge is how to leverage temporal information to achieve better segmentation. Previous efforts have primarily focused on pixel-level static-dynamic contexts matching, utilizing techniques such as optical flow and attention mechanisms. Instead, this paper rethinks static-dynamic contexts at the class level and proposes a novel static-dynamic class-level perceptual consistency (SD-CPC) framework. In this framework, we propose multivariate class prototype with contrastive learning and a static-dynamic semantic alignment module. The former provides class-level constraints for the model, obtaining personalized inter-class features and diversified intra-class features. The latter first establishes intra-frame spatial multi-scale and multi-level correlations to achieve static semantic alignment. Then, based on cross-frame static perceptual differences, it performs two-stage cross-frame selective aggregation to achieve dynamic semantic alignment. Meanwhile, we propose a window-based attention map calculation method that leverages the sparsity of attention points during cross-frame aggregation to reduce computation cost. Extensive experiments on VSPW and Cityscapes datasets show that the proposed approach outperforms state-of-the-art methods. Our implementation will be open-sourced on GitHub.