Action Duration Prediction for Segment-Level Alignment of Weakly-Labeled Videos

TL;DR

This paper introduces a novel Duration Network and Segment-Level Beam Search for weakly-supervised action alignment, improving robustness and achieving state-of-the-art results on benchmark datasets.

Contribution

The paper presents a new Duration Network and a Segment-Level Beam Search method for more accurate weakly-supervised action alignment.

Findings

More robust alignments for long videos

State-of-the-art results on Breakfast and Hollywood Extended datasets

Effective prediction of action durations at segment level

Abstract

This paper focuses on weakly-supervised action alignment, where only the ordered sequence of video-level actions is available for training. We propose a novel Duration Network, which captures a short temporal window of the video and learns to predict the remaining duration of a given action at any point in time with a level of granularity based on the type of that action. Further, we introduce a Segment-Level Beam Search to obtain the best alignment, that maximizes our posterior probability. Segment-Level Beam Search efficiently aligns actions by considering only a selected set of frames that have more confident predictions. The experimental results show that our alignments for long videos are more robust than existing models. Moreover, the proposed method achieves state of the art results in certain cases on the popular Breakfast and Hollywood Extended datasets.