Skeleton based Zero Shot Action Recognition in Joint Pose-Language Semantic Space

TL;DR

This paper introduces a zero shot action recognition method using joint pose-language semantic space, enabling the classification of unseen actions based on pose features and natural language descriptions, demonstrated on NTU RGB-D dataset.

Contribution

The work presents a novel pose-based zero shot action recognition network that jointly models visual pose features and natural language semantics for unseen action classification.

Findings

Effective recognition of unseen actions in NTU RGB-D dataset

Joint pose-language semantic space encodes useful knowledge

Model outperforms baseline methods in zero shot setting

Abstract

How does one represent an action? How does one describe an action that we have never seen before? Such questions are addressed by the Zero Shot Learning paradigm, where a model is trained on only a subset of classes and is evaluated on its ability to correctly classify an example from a class it has never seen before. In this work, we present a body pose based zero shot action recognition network and demonstrate its performance on the NTU RGB-D dataset. Our model learns to jointly encapsulate visual similarities based on pose features of the action performer as well as similarities in the natural language descriptions of the unseen action class names. We demonstrate how this pose-language semantic space encodes knowledge which allows our model to correctly predict actions not seen during training.