Skeleton-DML: Deep Metric Learning for Skeleton-Based One-Shot Action Recognition

TL;DR

This paper introduces Skeleton-DML, a deep metric learning approach for one-shot action recognition using a novel skeleton image representation, achieving state-of-the-art results on NTU RGB+D 120 dataset.

Contribution

It proposes a new image-based skeleton representation and a deep metric learning framework for one-shot action recognition, improving accuracy over existing methods.

Findings

Lifted state-of-the-art by 3.3% on NTU RGB+D 120 dataset.

Additional augmentation improved results by over 7.7%.

Demonstrated effectiveness of the proposed skeleton representation.

Abstract

One-shot action recognition allows the recognition of human-performed actions with only a single training example. This can influence human-robot-interaction positively by enabling the robot to react to previously unseen behaviour. We formulate the one-shot action recognition problem as a deep metric learning problem and propose a novel image-based skeleton representation that performs well in a metric learning setting. Therefore, we train a model that projects the image representations into an embedding space. In embedding space the similar actions have a low euclidean distance while dissimilar actions have a higher distance. The one-shot action recognition problem becomes a nearest-neighbor search in a set of activity reference samples. We evaluate the performance of our proposed representation against a variety of other skeleton-based image representations. In addition, we present an ablation study that shows the influence of different embedding vector sizes, losses and augmentation. Our approach lifts the state-of-the-art by 3.3% for the one-shot action recognition protocol on the NTU RGB+D 120 dataset under a comparable training setup. With additional augmentation our result improved over 7.7%.