WSLLN: Weakly Supervised Natural Language Localization Networks

TL;DR

WSLLN introduces a weakly supervised approach for language-based event localization in videos, significantly reducing annotation costs by learning from video-sentence pairs without needing explicit temporal annotations.

Contribution

The paper presents WSLLN, a novel end-to-end network that localizes events in videos using only weak supervision, outperforming existing methods on benchmark datasets.

Findings

Achieves state-of-the-art results on ActivityNet Captions

Reduces annotation effort by eliminating the need for temporal labels

Demonstrates effective segment-text matching in weakly supervised setting

Abstract

We propose weakly supervised language localization networks (WSLLN) to detect events in long, untrimmed videos given language queries. To learn the correspondence between visual segments and texts, most previous methods require temporal coordinates (start and end times) of events for training, which leads to high costs of annotation. WSLLN relieves the annotation burden by training with only video-sentence pairs without accessing to temporal locations of events. With a simple end-to-end structure, WSLLN measures segment-text consistency and conducts segment selection (conditioned on the text) simultaneously. Results from both are merged and optimized as a video-sentence matching problem. Experiments on ActivityNet Captions and DiDeMo demonstrate that WSLLN achieves state-of-the-art performance.