STVGFormer: Spatio-Temporal Video Grounding with Static-Dynamic Cross-Modal Understanding

TL;DR

STVGFormer is a novel framework for human-centric spatio-temporal video grounding that models static and dynamic visual-linguistic dependencies using cross-modal transformers, achieving state-of-the-art results.

Contribution

It introduces a static-dynamic cross-modal transformer framework with a novel interaction block for improved video grounding performance.

Findings

Achieved 39.6% vIoU on the HC-STVG track.

Won first place in the 4th Person in Context Challenge.

Effectively models intra-frame and inter-frame dependencies.

Abstract

In this technical report, we introduce our solution to human-centric spatio-temporal video grounding task. We propose a concise and effective framework named STVGFormer, which models spatiotemporal visual-linguistic dependencies with a static branch and a dynamic branch. The static branch performs cross-modal understanding in a single frame and learns to localize the target object spatially according to intra-frame visual cues like object appearances. The dynamic branch performs cross-modal understanding across multiple frames. It learns to predict the starting and ending time of the target moment according to dynamic visual cues like motions. Both the static and dynamic branches are designed as cross-modal transformers. We further design a novel static-dynamic interaction block to enable the static and dynamic branches to transfer useful and complementary information from each other, which is shown to be effective to improve the prediction on hard cases. Our proposed method achieved 39.6% vIoU and won the first place in the HC-STVG track of the 4th Person in Context Challenge.