Telling Stories for Common Sense Zero-Shot Action Recognition

TL;DR

This paper introduces a new dataset called Stories with detailed textual narratives for action classes, enabling improved zero-shot video action recognition without fine-tuning, and achieves state-of-the-art results on benchmarks.

Contribution

The paper presents the Stories dataset with rich textual descriptions and a novel method leveraging this data to enhance zero-shot action recognition performance.

Findings

Achieved up to 6.1% improvement in top-1 accuracy on benchmarks.

Introduced a new dataset with multi-sentence narratives for action classes.

Demonstrated the effectiveness of textual context in zero-shot transfer.

Abstract

Video understanding has long suffered from reliance on large labeled datasets, motivating research into zero-shot learning. Recent progress in language modeling presents opportunities to advance zero-shot video analysis, but constructing an effective semantic space relating action classes remains challenging. We address this by introducing a novel dataset, Stories, which contains rich textual descriptions for diverse action classes extracted from WikiHow articles. For each class, we extract multi-sentence narratives detailing the necessary steps, scenes, objects, and verbs that characterize the action. This contextual data enables modeling of nuanced relationships between actions, paving the way for zero-shot transfer. We also propose an approach that harnesses Stories to improve feature generation for training zero-shot classification. Without any target dataset fine-tuning, our method achieves new state-of-the-art on multiple benchmarks, improving top-1 accuracy by up to 6.1%. We believe Stories provides a valuable resource that can catalyze progress in zero-shot action recognition. The textual narratives forge connections between seen and unseen classes, overcoming the bottleneck of labeled data that has long impeded advancements in this exciting domain. The data can be found here: https://github.com/kini5gowda/Stories .