From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation

TL;DR

This paper introduces ARR, an attention-based end-to-end video model that improves action anticipation by decomposing the task into recognition and reasoning, leveraging statistical action relationships and unsupervised pre-training.

Contribution

The paper proposes ARR, a novel architecture that combines recognition and sequence reasoning for better action anticipation, including an unsupervised pre-training approach for the decoder.

Findings

ARR outperforms existing methods on multiple datasets.

Unsupervised pre-training enhances reasoning capabilities.

Effective modeling of action relationships improves prediction accuracy.

Abstract

The action anticipation task refers to predicting what action will happen based on observed videos, which requires the model to have a strong ability to summarize the present and then reason about the future. Experience and common sense suggest that there is a significant correlation between different actions, which provides valuable prior knowledge for the action anticipation task. However, previous methods have not effectively modeled this underlying statistical relationship. To address this issue, we propose a novel end-to-end video modeling architecture that utilizes attention mechanisms, named Anticipation via Recognition and Reasoning (ARR). ARR decomposes the action anticipation task into action recognition and sequence reasoning tasks, and effectively learns the statistical relationship between actions by next action prediction (NAP). In comparison to existing temporal aggregation strategies, ARR is able to extract more effective features from observable videos to make more reasonable predictions. In addition, to address the challenge of relationship modeling that requires extensive training data, we propose an innovative approach for the unsupervised pre-training of the decoder, which leverages the inherent temporal dynamics of video to enhance the reasoning capabilities of the network. Extensive experiments on the Epic-kitchen-100, EGTEA Gaze+, and 50salads datasets demonstrate the efficacy of the proposed methods. The code is available at https://github.com/linuxsino/ARR.