Human-centric Spatio-Temporal Video Grounding via the Combination of Mutual Matching Network and TubeDETR

TL;DR

This paper presents a human-centric spatio-temporal video grounding method combining TubeDETR and Mutual Matching Network, achieving third place in a challenge by effectively localizing persons in videos based on text descriptions.

Contribution

The paper introduces a novel combination of TubeDETR and MMN for improved spatio-temporal grounding of persons in videos, integrating spatial and temporal localization.

Findings

Achieved third place in the 4th PIC challenge.

Effectively combines spatial and temporal localization.

Improved accuracy in person grounding in videos.

Abstract

In this technical report, we represent our solution for the Human-centric Spatio-Temporal Video Grounding (HC-STVG) track of the 4th Person in Context (PIC) workshop and challenge. Our solution is built on the basis of TubeDETR and Mutual Matching Network (MMN). Specifically, TubeDETR exploits a video-text encoder and a space-time decoder to predict the starting time, the ending time and the tube of the target person. MMN detects persons in images, links them as tubes, extracts features of person tubes and the text description, and predicts the similarities between them to choose the most likely person tube as the grounding result. Our solution finally finetunes the results by combining the spatio localization of MMN and with temporal localization of TubeDETR. In the HC-STVG track of the 4th PIC challenge, our solution achieves the third place.