Deeply Interleaved Two-Stream Encoder for Referring Video Segmentation

TL;DR

This paper introduces a novel deep interleaved two-stream encoder with a vision-language mutual guidance module and a language-guided multi-scale dynamic filtering module, significantly improving referring video segmentation by better multi-modal fusion and temporal coherence.

Contribution

It proposes a new hierarchical two-stream encoder with mutual guidance and dynamic filtering modules for enhanced multi-modal fusion and temporal alignment in referring video segmentation.

Findings

Outperforms existing methods on four benchmark datasets.

Demonstrates improved multi-modal feature fusion and temporal coherence.

Achieves state-of-the-art segmentation accuracy.

Abstract

Referring video segmentation aims to segment the corresponding video object described by the language expression. To address this task, we first design a two-stream encoder to extract CNN-based visual features and transformer-based linguistic features hierarchically, and a vision-language mutual guidance (VLMG) module is inserted into the encoder multiple times to promote the hierarchical and progressive fusion of multi-modal features. Compared with the existing multi-modal fusion methods, this two-stream encoder takes into account the multi-granularity linguistic context, and realizes the deep interleaving between modalities with the help of VLGM. In order to promote the temporal alignment between frames, we further propose a language-guided multi-scale dynamic filtering (LMDF) module to strengthen the temporal coherence, which uses the language-guided spatial-temporal features to generate a set of position-specific dynamic filters to more flexibly and effectively update the feature of current frame. Extensive experiments on four datasets verify the effectiveness of the proposed model.