Memory-and-Anticipation Transformer for Online Action Understanding

TL;DR

This paper introduces the Memory-and-Anticipation Transformer (MAT), a novel approach that models entire temporal structures for online action detection and anticipation, outperforming existing memory-based methods on multiple benchmarks.

Contribution

The paper proposes a new memory-anticipation paradigm and the MAT model, enabling unified processing of online action detection and anticipation tasks with superior performance.

Findings

Outperforms existing methods on four benchmarks

Effectively models entire temporal structures including past, present, and future

Unifies online action detection and anticipation tasks

Abstract

Most existing forecasting systems are memory-based methods, which attempt to mimic human forecasting ability by employing various memory mechanisms and have progressed in temporal modeling for memory dependency. Nevertheless, an obvious weakness of this paradigm is that it can only model limited historical dependence and can not transcend the past. In this paper, we rethink the temporal dependence of event evolution and propose a novel memory-anticipation-based paradigm to model an entire temporal structure, including the past, present, and future. Based on this idea, we present Memory-and-Anticipation Transformer (MAT), a memory-anticipation-based approach, to address the online action detection and anticipation tasks. In addition, owing to the inherent superiority of MAT, it can process online action detection and anticipation tasks in a unified manner. The proposed MAT model is tested on four challenging benchmarks TVSeries, THUMOS'14, HDD, and EPIC-Kitchens-100, for online action detection and anticipation tasks, and it significantly outperforms all existing methods. Code is available at https://github.com/Echo0125/Memory-and-Anticipation-Transformer.