Spatial Transformer Network with Transfer Learning for Small-scale Fine-grained Skeleton-based Tai Chi Action Recognition

TL;DR

This paper introduces a transfer learning approach using a spatial transformer network to improve small-scale, fine-grained Tai Chi action recognition from skeleton data, achieving state-of-the-art accuracy.

Contribution

It proposes a novel transfer learning method with a Transformer-based network pre-trained on large-scale data for fine-grained Tai Chi action recognition.

Findings

High accuracy achieved on small-scale Tai Chi dataset

Effective transfer learning from large-scale datasets

State-of-the-art performance compared to previous methods

Abstract

Human action recognition is a quite hugely investigated area where most remarkable action recognition networks usually use large-scale coarse-grained action datasets of daily human actions as inputs to state the superiority of their networks. We intend to recognize our small-scale fine-grained Tai Chi action dataset using neural networks and propose a transfer-learning method using NTU RGB+D dataset to pre-train our network. More specifically, the proposed method first uses a large-scale NTU RGB+D dataset to pre-train the Transformer-based network for action recognition to extract common features among human motion. Then we freeze the network weights except for the fully connected (FC) layer and take our Tai Chi actions as inputs only to train the initialized FC weights. Experimental results show that our general model pipeline can reach a high accuracy of small-scale fine-grained Tai Chi action recognition with even few inputs and demonstrate that our method achieves the state-of-the-art performance compared with previous Tai Chi action recognition methods.