Bridging the gap between Human Action Recognition and Online Action Detection

TL;DR

This paper introduces OKDAD, a novel teacher-student framework for online action detection that improves feature extraction and achieves state-of-the-art results on infrared datasets without future knowledge.

Contribution

We propose a new teacher-student training strategy with layer reuse and cosine similarity for task-specific feature extraction in online action detection.

Findings

State-of-the-art results on NTU RGB+D and PKU MMD datasets.

First use of infrared data from RGB-D cameras for online action detection.

Significant baseline improvements with our feature learning approach.

Abstract

Action recognition, early prediction, and online action detection are complementary disciplines that are often studied independently. Most online action detection networks use a pre-trained feature extractor, which might not be optimal for its new task. We address the task-specific feature extraction with a teacher-student framework between the aforementioned disciplines, and a novel training strategy. Our network, Online Knowledge Distillation Action Detection network (OKDAD), embeds online early prediction and online temporal segment proposal subnetworks in parallel. Low interclass and high intraclass similarity are encouraged during teacher training. Knowledge distillation to the OKDAD network is ensured via layer reuse and cosine similarity between teacher-student feature vectors. Layer reuse and similarity learning significantly improve our baseline which uses a generic feature extractor. We evaluate our framework on infrared videos from two popular datasets, NTU RGB+D (action recognition, early prediction) and PKU MMD (action detection). Unlike previous attempts on those datasets, our student networks perform without any knowledge of the future. Even with this added difficulty, we achieve state-of-the-art results on both datasets. Moreover, our networks use infrared from RGB-D cameras, which we are the first to use for online action detection, to our knowledge.