TagBook: A Semantic Video Representation without Supervision for Event Detection

TL;DR

TagBook introduces a novel, supervision-free semantic video representation using social tags for event detection, outperforming supervised methods in few- and zero-example scenarios.

Contribution

It proposes a new tag propagation-based video representation that does not require training concept detectors, enabling effective event detection with minimal or no labeled data.

Findings

Outperforms state-of-the-art supervised methods in zero- and few-example detection

Effective on multiple datasets including TRECVID and Columbia Video Dataset

Simple algorithm with competitive performance

Abstract

We consider the problem of event detection in video for scenarios where only few, or even zero examples are available for training. For this challenging setting, the prevailing solutions in the literature rely on a semantic video representation obtained from thousands of pre-trained concept detectors. Different from existing work, we propose a new semantic video representation that is based on freely available social tagged videos only, without the need for training any intermediate concept detectors. We introduce a simple algorithm that propagates tags from a video's nearest neighbors, similar in spirit to the ones used for image retrieval, but redesign it for video event detection by including video source set refinement and varying the video tag assignment. We call our approach TagBook and study its construction, descriptiveness and detection performance on the TRECVID 2013 and 2014 multimedia event detection datasets and the Columbia Consumer Video dataset. Despite its simple nature, the proposed TagBook video representation is remarkably effective for few-example and zero-example event detection, even outperforming very recent state-of-the-art alternatives building on supervised representations.