Zero-shot Natural Language Video Localization

TL;DR

This paper introduces a zero-shot approach for natural language video localization that leverages unpaired data and pseudo-supervision, reducing the need for costly annotations and outperforming some supervised methods.

Contribution

It presents the first zero-shot training method for video localization using pseudo-supervision from unpaired data, eliminating annotation costs.

Findings

Outperforms baseline methods on Charades-STA and ActivityNet-Captions datasets.

Uses unpaired text, videos, and object detection to train without annotations.

Achieves competitive results compared to supervised approaches.

Abstract

Understanding videos to localize moments with natural language often requires large expensive annotated video regions paired with language queries. To eliminate the annotation costs, we make a first attempt to train a natural language video localization model in zero-shot manner. Inspired by unsupervised image captioning setup, we merely require random text corpora, unlabeled video collections, and an off-the-shelf object detector to train a model. With the unpaired data, we propose to generate pseudo-supervision of candidate temporal regions and corresponding query sentences, and develop a simple NLVL model to train with the pseudo-supervision. Our empirical validations show that the proposed pseudo-supervised method outperforms several baseline approaches and a number of methods using stronger supervision on Charades-STA and ActivityNet-Captions.