STARS: Self-supervised Tuning for 3D Action Recognition in Skeleton Sequences

TL;DR

STARS introduces a self-supervised tuning method combining masked prediction and contrastive learning to improve 3D skeleton-based action recognition, especially in few-shot scenarios, achieving state-of-the-art results.

Contribution

The paper proposes a novel self-supervised tuning approach that enhances clustering and generalization in skeleton-based action recognition without relying on hand-crafted augmentations.

Findings

Achieves state-of-the-art results on NTU-60, NTU-120, and PKU-MMD benchmarks.

Significantly improves few-shot action recognition performance.

Outperforms masked prediction models in generalization tasks.

Abstract

Self-supervised pretraining methods with masked prediction demonstrate remarkable within-dataset performance in skeleton-based action recognition. However, we show that, unlike contrastive learning approaches, they do not produce well-separated clusters. Additionally, these methods struggle with generalization in few-shot settings. To address these issues, we propose Self-supervised Tuning for 3D Action Recognition in Skeleton sequences (STARS). Specifically, STARS first uses a masked prediction stage using an encoder-decoder architecture. It then employs nearest-neighbor contrastive learning to partially tune the weights of the encoder, enhancing the formation of semantic clusters for different actions. By tuning the encoder for a few epochs, and without using hand-crafted data augmentations, STARS achieves state-of-the-art self-supervised results in various benchmarks, including NTU-60, NTU-120, and PKU-MMD. In addition, STARS exhibits significantly better results than masked prediction models in few-shot settings, where the model has not seen the actions throughout pretraining. Project page: https://soroushmehraban.github.io/stars/