Long-Term Anticipation of Activities with Cycle Consistency

TL;DR

This paper introduces an end-to-end framework for long-term activity anticipation in videos, utilizing cycle consistency loss to improve prediction accuracy over extended time horizons, achieving state-of-the-art results.

Contribution

The paper proposes a novel cycle consistency loss for directly predicting future activities from observed frames in an end-to-end manner, extending anticipation capabilities.

Findings

Achieves state-of-the-art results on Breakfast and 50Salads datasets.

Demonstrates improved long-term activity prediction accuracy.

Validates effectiveness of cycle consistency loss in activity anticipation.

Abstract

With the success of deep learning methods in analyzing activities in videos, more attention has recently been focused towards anticipating future activities. However, most of the work on anticipation either analyzes a partially observed activity or predicts the next action class. Recently, new approaches have been proposed to extend the prediction horizon up to several minutes in the future and that anticipate a sequence of future activities including their durations. While these works decouple the semantic interpretation of the observed sequence from the anticipation task, we propose a framework for anticipating future activities directly from the features of the observed frames and train it in an end-to-end fashion. Furthermore, we introduce a cycle consistency loss over time by predicting the past activities given the predicted future. Our framework achieves state-of-the-art results on two datasets: the Breakfast dataset and 50Salads.