Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models

TL;DR

This paper presents a novel zero-shot video semantic segmentation method leveraging pre-trained diffusion models, incorporating temporal modeling and refinement strategies to achieve high-quality, consistent segmentation without training on video data.

Contribution

The authors introduce a new zero-shot VSS framework based on pre-trained diffusion models, with a scene context model and temporal refinement, outperforming existing methods without training.

Findings

Outperforms existing zero-shot image segmentation on VSS benchmarks

Rivals supervised VSS approaches on VSPW dataset

Achieves high-quality, temporally consistent segmentation without training

Abstract

We introduce the first zero-shot approach for Video Semantic Segmentation (VSS) based on pre-trained diffusion models. A growing research direction attempts to employ diffusion models to perform downstream vision tasks by exploiting their deep understanding of image semantics. Yet, the majority of these approaches have focused on image-related tasks like semantic correspondence and segmentation, with less emphasis on video tasks such as VSS. Ideally, diffusion-based image semantic segmentation approaches can be applied to videos in a frame-by-frame manner. However, we find their performance on videos to be subpar due to the absence of any modeling of temporal information inherent in the video data. To this end, we tackle this problem and introduce a framework tailored for VSS based on pre-trained image and video diffusion models. We propose building a scene context model based on the diffusion features, where the model is autoregressively updated to adapt to scene changes. This context model predicts per-frame coarse segmentation maps that are temporally consistent. To refine these maps further, we propose a correspondence-based refinement strategy that aggregates predictions temporally, resulting in more confident predictions. Finally, we introduce a masked modulation approach to upsample the coarse maps to the full resolution at a high quality. Experiments show that our proposed approach outperforms existing zero-shot image semantic segmentation approaches significantly on various VSS benchmarks without any training or fine-tuning. Moreover, it rivals supervised VSS approaches on the VSPW dataset despite not being explicitly trained for VSS.