Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions

TL;DR

This paper introduces a hierarchical model that transfers knowledge from large-scale text data to enable zero-shot recognition and prediction of procedural actions in videos, demonstrated on a new recipe dataset.

Contribution

The authors propose a novel hierarchical model that leverages text-based instructional knowledge for zero-shot video action recognition and anticipation.

Findings

Effective zero-shot recognition and prediction of unseen actions.

Model generalizes well with limited video training data.

Introduces the Tasty Videos Dataset V2 for benchmarking.

Abstract

Can we teach a robot to recognize and make predictions for activities that it has never seen before? We tackle this problem by learning models for video from text. This paper presents a hierarchical model that generalizes instructional knowledge from large-scale text corpora and transfers the knowledge to video. Given a portion of an instructional video, our model recognizes and predicts coherent and plausible actions multiple steps into the future, all in rich natural language. To demonstrate the capabilities of our model, we introduce the \emph{Tasty Videos Dataset V2}, a collection of 4022 recipes for zero-shot learning, recognition and anticipation. Extensive experiments with various evaluation metrics demonstrate the potential of our method for generalization, given limited video data for training models.