Language-free Training for Zero-shot Video Grounding

TL;DR

This paper introduces a novel language-free training framework for zero-shot video grounding that leverages visual features and CLIP's aligned visual-language space, eliminating the need for annotated language data.

Contribution

It proposes a new training method that learns video grounding without language annotations by selecting temporal intervals and using visual features as pseudo-language, outperforming existing methods.

Findings

Outperforms existing zero-shot video grounding methods.

Surpasses several weakly-supervised approaches.

Demonstrates effectiveness on standard datasets.

Abstract

Given an untrimmed video and a language query depicting a specific temporal moment in the video, video grounding aims to localize the time interval by understanding the text and video simultaneously. One of the most challenging issues is an extremely time- and cost-consuming annotation collection, including video captions in a natural language form and their corresponding temporal regions. In this paper, we present a simple yet novel training framework for video grounding in the zero-shot setting, which learns a network with only video data without any annotation. Inspired by the recent language-free paradigm, i.e. training without language data, we train the network without compelling the generation of fake (pseudo) text queries into a natural language form. Specifically, we propose a method for learning a video grounding model by selecting a temporal interval as a hypothetical correct answer and considering the visual feature selected by our method in the interval as a language feature, with the help of the well-aligned visual-language space of CLIP. Extensive experiments demonstrate the prominence of our language-free training framework, outperforming the existing zero-shot video grounding method and even several weakly-supervised approaches with large margins on two standard datasets.