Weakly Supervised Action Labeling in Videos Under Ordering Constraints

TL;DR

This paper introduces a weakly supervised method for localizing actions in videos using ordering constraints from text annotations, jointly learning classifiers and temporal labels.

Contribution

It formulates action localization as a weakly supervised temporal assignment problem with ordering constraints, enabling joint learning of classifiers and temporal labels.

Findings

Effective localization of actions in videos with minimal supervision

Successful application to a large dataset of Hollywood movies

Joint learning improves classifier discriminability

Abstract

We are given a set of video clips, each one annotated with an {\em ordered} list of actions, such as "walk" then "sit" then "answer phone" extracted from, for example, the associated text script. We seek to temporally localize the individual actions in each clip as well as to learn a discriminative classifier for each action. We formulate the problem as a weakly supervised temporal assignment with ordering constraints. Each video clip is divided into small time intervals and each time interval of each video clip is assigned one action label, while respecting the order in which the action labels appear in the given annotations. We show that the action label assignment can be determined together with learning a classifier for each action in a discriminative manner. We evaluate the proposed model on a new and challenging dataset of 937 video clips with a total of 787720 frames containing sequences of 16 different actions from 69 Hollywood movies.