Video-Guided Curriculum Learning for Spoken Video Grounding

TL;DR

This paper introduces a new task called spoken video grounding, utilizing a novel curriculum learning approach that leverages visual cues to improve understanding of noisy spoken language in videos.

Contribution

The paper proposes a video-guided curriculum learning method for spoken video grounding and creates the first large-scale dataset for this task, enhancing model robustness in noisy environments.

Findings

VGCL improves pre-training efficiency and grounding accuracy.

Model outperforms ASR-based methods in noisy conditions.

Introduces the ActivityNet Speech dataset for large-scale evaluation.

Abstract

In this paper, we introduce a new task, spoken video grounding (SVG), which aims to localize the desired video fragments from spoken language descriptions. Compared with using text, employing audio requires the model to directly exploit the useful phonemes and syllables related to the video from raw speech. Moreover, we randomly add environmental noises to this speech audio, further increasing the difficulty of this task and better simulating real applications. To rectify the discriminative phonemes and extract video-related information from noisy audio, we develop a novel video-guided curriculum learning (VGCL) during the audio pre-training process, which can make use of the vital visual perceptions to help understand the spoken language and suppress the external noise. Considering during inference the model can not obtain ground truth video segments, we design a curriculum strategy that gradually shifts the input video from the ground truth to the entire video content during pre-training. Finally, the model can learn how to extract critical visual information from the entire video clip to help understand the spoken language. In addition, we collect the first large-scale spoken video grounding dataset based on ActivityNet, which is named as ActivityNet Speech dataset. Extensive experiments demonstrate our proposed video-guided curriculum learning can facilitate the pre-training process to obtain a mutual audio encoder, significantly promoting the performance of spoken video grounding tasks. Moreover, we prove that in the case of noisy sound, our model outperforms the method that grounding video with ASR transcripts, further demonstrating the effectiveness of our curriculum strategy.