Action Anticipation with RBF Kernelized Feature Mapping RNN

TL;DR

This paper presents a novel RNN architecture using RBF kernels for future video feature generation and action anticipation, achieving significant improvements over previous methods on multiple datasets.

Contribution

Introduces a feature mapping RNN with parameter sharing, RBF kernels, and adversarial training for efficient future feature generation in action anticipation.

Findings

18% improvement on JHMDB-21 dataset

6% improvement on UCF101-24 dataset

13% improvement on UT-Interaction dataset

Abstract

We introduce a novel Recurrent Neural Network-based algorithm for future video feature generation and action anticipation called feature mapping RNN. Our novel RNN architecture builds upon three effective principles of machine learning, namely parameter sharing, Radial Basis Function kernels and adversarial training. Using only some of the earliest frames of a video, the feature mapping RNN is able to generate future features with a fraction of the parameters needed in traditional RNN. By feeding these future features into a simple multi-layer perceptron facilitated with an RBF kernel layer, we are able to accurately predict the action in the video. In our experiments, we obtain 18% improvement on JHMDB-21 dataset, 6% on UCF101-24 and 13% improvement on UT-Interaction datasets over prior state-of-the-art for action anticipation.