Learning Scene Flow With Skeleton Guidance For 3D Action Recognition

TL;DR

This paper introduces a novel deep learning approach for 3D action recognition that leverages skeleton-guided attention on 3D flow data, significantly improving accuracy on a large, challenging dataset.

Contribution

It proposes a skeleton-guided spatial attention mechanism and an extended deep skeleton model to enhance 3D flow-based action recognition.

Findings

Achieves state-of-the-art results on NTU RGB+D dataset

Effectively emphasizes motion features near body joints

Demonstrates robustness against noise in 3D flow data

Abstract

Among the existing modalities for 3D action recognition, 3D flow has been poorly examined, although conveying rich motion information cues for human actions. Presumably, its susceptibility to noise renders it intractable, thus challenging the learning process within deep models. This work demonstrates the use of 3D flow sequence by a deep spatiotemporal model and further proposes an incremental two-level spatial attention mechanism, guided from skeleton domain, for emphasizing motion features close to the body joint areas and according to their informativeness. Towards this end, an extended deep skeleton model is also introduced to learn the most discriminant action motion dynamics, so as to estimate an informativeness score for each joint. Subsequently, a late fusion scheme is adopted between the two models for learning the high level cross-modal correlations. Experimental results on the currently largest and most challenging dataset NTU RGB+D, demonstrate the effectiveness of the proposed approach, achieving state-of-the-art results.