A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects

TL;DR

This survey comprehensively reviews recent advances, challenges, and future prospects in Video Scene Parsing, emphasizing deep learning methods, technical hurdles, and benchmarking standards across various vision tasks.

Contribution

It provides a holistic analysis of VSP evolution, compares datasets and metrics, and discusses emerging trends and research directions in the field.

Findings

Deep learning has significantly advanced VSP performance.

Maintaining temporal consistency remains a key challenge.

Transformer-based architectures are increasingly prominent in VSP.

Abstract

Video Scene Parsing (VSP) has emerged as a cornerstone in computer vision, facilitating the simultaneous segmentation, recognition, and tracking of diverse visual entities in dynamic scenes. In this survey, we present a holistic review of recent advances in VSP, covering a wide array of vision tasks, including Video Semantic Segmentation (VSS), Video Instance Segmentation (VIS), Video Panoptic Segmentation (VPS), as well as Video Tracking and Segmentation (VTS), and Open-Vocabulary Video Segmentation (OVVS). We systematically analyze the evolution from traditional hand-crafted features to modern deep learning paradigms -- spanning from fully convolutional networks to the latest transformer-based architectures -- and assess their effectiveness in capturing both local and global temporal contexts. Furthermore, our review critically discusses the technical challenges, ranging from maintaining temporal consistency to handling complex scene dynamics, and offers a comprehensive comparative study of datasets and evaluation metrics that have shaped current benchmarking standards. By distilling the key contributions and shortcomings of state-of-the-art methodologies, this survey highlights emerging trends and prospective research directions that promise to further elevate the robustness and adaptability of VSP in real-world applications.