Weakly supervised learning of actions from transcripts

TL;DR

This paper introduces a weakly supervised learning method for human actions in videos using only transcripts, enabling action localization and classification without frame-level annotations, and demonstrates competitive results across multiple datasets.

Contribution

The authors propose a novel approach that infers action models from transcript sequences, eliminating the need for detailed frame annotations and improving transcript-video alignment.

Findings

Achieves competitive action localization and classification

Outperforms state-of-the-art transcript alignment methods

Effective across diverse activity datasets

Abstract

We present an approach for weakly supervised learning of human actions from video transcriptions. Our system is based on the idea that, given a sequence of input data and a transcript, i.e. a list of the order the actions occur in the video, it is possible to infer the actions within the video stream, and thus, learn the related action models without the need for any frame-based annotation. Starting from the transcript information at hand, we split the given data sequences uniformly based on the number of expected actions. We then learn action models for each class by maximizing the probability that the training video sequences are generated by the action models given the sequence order as defined by the transcripts. The learned model can be used to temporally segment an unseen video with or without transcript. We evaluate our approach on four distinct activity datasets, namely Hollywood Extended, MPII Cooking, Breakfast and CRIM13. We show that our system is able to align the scripted actions with the video data and that the learned models localize and classify actions competitively in comparison to models trained with full supervision, i.e. with frame level annotations, and that they outperform any current state-of-the-art approach for aligning transcripts with video data.