Temporal Context Consistency Above All: Enhancing Long-Term Anticipation by Learning and Enforcing Temporal Constraints

TL;DR

This paper introduces a novel approach for long-term action anticipation in videos, emphasizing temporal context consistency through a bi-directional regularizer and transition modeling, leading to improved predictions on multiple benchmarks.

Contribution

It proposes a bi-directional temporal regularizer and a transition matrix for better long-term action prediction, enhancing temporal coherence and sequence modeling.

Findings

Achieves superior or comparable results on four benchmark datasets.

Demonstrates the effectiveness of temporal regularization and transition modeling.

Outperforms existing probabilistic and LLM-based methods.

Abstract

This paper proposes a method for long-term action anticipation (LTA), the task of predicting action labels and their duration in a video given the observation of an initial untrimmed video interval. We build on an encoder-decoder architecture with parallel decoding and make two key contributions. First, we introduce a bi-directional action context regularizer module on the top of the decoder that ensures temporal context coherence in temporally adjacent segments. Second, we learn from classified segments a transition matrix that models the probability of transitioning from one action to another and the sequence is optimized globally over the full prediction interval. In addition, we use a specialized encoder for the task of action segmentation to increase the quality of the predictions in the observation interval at inference time, leading to a better understanding of the past. We validate our methods on four benchmark datasets for LTA, the EpicKitchen-55, EGTEA+, 50Salads and Breakfast demonstrating superior or comparable performance to state-of-the-art methods, including probabilistic models and also those based on Large Language Models, that assume trimmed video as input. The code will be released upon acceptance.