Adversarial Memory Networks for Action Prediction

TL;DR

This paper introduces adversarial memory networks (AMemNet) for action prediction, which generate full video features from partial observations using a memory generator and class-aware discriminator, improving prediction accuracy.

Contribution

The paper proposes a novel adversarial memory network with key-value structured memory and class-aware discriminator for improved action prediction from partial videos.

Findings

AMemNet outperforms state-of-the-art methods on UCF-101 and HMDB51 datasets.

The memory generator effectively models partial-to-full video feature mapping.

Adversarial training enhances the realism and discriminability of generated features.

Abstract

Action prediction aims to infer the forthcoming human action with partially-observed videos, which is a challenging task due to the limited information underlying early observations. Existing methods mainly adopt a reconstruction strategy to handle this task, expecting to learn a single mapping function from partial observations to full videos to facilitate the prediction process. In this study, we propose adversarial memory networks (AMemNet) to generate the "full video" feature conditioning on a partial video query from two new aspects. Firstly, a key-value structured memory generator is designed to memorize different partial videos as key memories and dynamically write full videos in value memories with gating mechanism and querying attention. Secondly, we develop a class-aware discriminator to guide the memory generator to deliver not only realistic but also discriminative full video features upon adversarial training. The final prediction result of AMemNet is given by late fusion over RGB and optical flow streams. Extensive experimental results on two benchmark video datasets, UCF-101 and HMDB51, are provided to demonstrate the effectiveness of the proposed AMemNet model over state-of-the-art methods.