Weakly-Supervised Action Detection Guided by Audio Narration

TL;DR

This paper introduces a weakly-supervised action detection model that leverages audio narration and multimodal features to reduce annotation costs in video datasets, achieving effective detection without detailed annotations.

Contribution

The paper presents a novel approach that uses narration supervision and multimodal features to perform action detection without expensive frame-level annotations.

Findings

Noisy audio narration effectively guides action detection.

Multimodal features improve detection accuracy.

Reduces reliance on detailed annotations.

Abstract

Videos are more well-organized curated data sources for visual concept learning than images. Unlike the 2-dimensional images which only involve the spatial information, the additional temporal dimension bridges and synchronizes multiple modalities. However, in most video detection benchmarks, these additional modalities are not fully utilized. For example, EPIC Kitchens is the largest dataset in first-person (egocentric) vision, yet it still relies on crowdsourced information to refine the action boundaries to provide instance-level action annotations. We explored how to eliminate the expensive annotations in video detection data which provide refined boundaries. We propose a model to learn from the narration supervision and utilize multimodal features, including RGB, motion flow, and ambient sound. Our model learns to attend to the frames related to the narration label while suppressing the irrelevant frames from being used. Our experiments show that noisy audio narration suffices to learn a good action detection model, thus reducing annotation expenses.