Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipation

TL;DR

This paper introduces a Gated Temporal Diffusion network for long-term action anticipation that models uncertainty in predictions, achieving state-of-the-art results on multiple datasets in both stochastic and deterministic scenarios.

Contribution

The paper proposes a novel Gated Temporal Diffusion network with a Gated Anticipation Network that jointly models observed and unobserved video frames, capturing uncertainty in long-term action prediction.

Findings

Achieves state-of-the-art results on Breakfast, Assembly101, and 50Salads datasets.

Effectively models uncertainty in both observed and future actions.

Performs well in both stochastic and deterministic settings.

Abstract

Long-term action anticipation has become an important task for many applications such as autonomous driving and human-robot interaction. Unlike short-term anticipation, predicting more actions into the future imposes a real challenge with the increasing uncertainty in longer horizons. While there has been a significant progress in predicting more actions into the future, most of the proposed methods address the task in a deterministic setup and ignore the underlying uncertainty. In this paper, we propose a novel Gated Temporal Diffusion (GTD) network that models the uncertainty of both the observation and the future predictions. As generator, we introduce a Gated Anticipation Network (GTAN) to model both observed and unobserved frames of a video in a mutual representation. On the one hand, using a mutual representation for past and future allows us to jointly model ambiguities in the observation and future, while on the other hand GTAN can by design treat the observed and unobserved parts differently and steer the information flow between them. Our model achieves state-of-the-art results on the Breakfast, Assembly101 and 50Salads datasets in both stochastic and deterministic settings. Code: https://github.com/olga-zats/GTDA .