Anticipative Video Transformer

TL;DR

The paper introduces Anticipative Video Transformer (AVT), a novel attention-based architecture that predicts future actions in videos by jointly modeling observed sequences and future frame features, achieving state-of-the-art results.

Contribution

AVT is a new end-to-end video transformer that anticipates future actions by attending to past observations and predicting future frame features, outperforming existing methods.

Findings

Achieves top performance on four action anticipation benchmarks.

Wins first place in the EpicKitchens-100 CVPR'21 challenge.

Effectively captures long-range dependencies for action prediction.

Abstract

We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions. We train the model jointly to predict the next action in a video sequence, while also learning frame feature encoders that are predictive of successive future frames' features. Compared to existing temporal aggregation strategies, AVT has the advantage of both maintaining the sequential progression of observed actions while still capturing long-range dependencies--both critical for the anticipation task. Through extensive experiments, we show that AVT obtains the best reported performance on four popular action anticipation benchmarks: EpicKitchens-55, EpicKitchens-100, EGTEA Gaze+, and 50-Salads; and it wins first place in the EpicKitchens-100 CVPR'21 challenge.