Mining Relations among Cross-Frame Affinities for Video Semantic Segmentation

TL;DR

This paper introduces a novel approach to video semantic segmentation by mining relations among cross-frame affinities, utilizing single-scale and multi-scale affinity refinement techniques to enhance temporal information aggregation.

Contribution

It proposes new methods, SAR and MAA, for refining and aggregating cross-frame affinities, along with a Selective Token Masking strategy to improve efficiency and effectiveness.

Findings

Outperforms state-of-the-art VSS methods

Effective affinity refinement and aggregation techniques

Improved temporal information utilization

Abstract

The essence of video semantic segmentation (VSS) is how to leverage temporal information for prediction. Previous efforts are mainly devoted to developing new techniques to calculate the cross-frame affinities such as optical flow and attention. Instead, this paper contributes from a different angle by mining relations among cross-frame affinities, upon which better temporal information aggregation could be achieved. We explore relations among affinities in two aspects: single-scale intrinsic correlations and multi-scale relations. Inspired by traditional feature processing, we propose Single-scale Affinity Refinement (SAR) and Multi-scale Affinity Aggregation (MAA). To make it feasible to execute MAA, we propose a Selective Token Masking (STM) strategy to select a subset of consistent reference tokens for different scales when calculating affinities, which also improves the efficiency of our method. At last, the cross-frame affinities strengthened by SAR and MAA are adopted for adaptively aggregating temporal information. Our experiments demonstrate that the proposed method performs favorably against state-of-the-art VSS methods. The code is publicly available at https://github.com/GuoleiSun/VSS-MRCFA