Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyond

TL;DR

This paper provides a comprehensive survey and benchmark of self-supervised learning methods for skeleton-based action recognition, introduces a novel SSL approach that enhances generalization across multiple tasks, and demonstrates its effectiveness through extensive experiments.

Contribution

It offers the first systematic review and benchmarking of skeleton SSL methods, and proposes a new multi-granularity SSL technique that improves generalization for various downstream tasks.

Findings

Most SSL methods rely on a single paradigm and are evaluated only on recognition.

The proposed SSL method significantly improves performance across recognition, retrieval, detection, and few-shot learning.

Extensive experiments on large datasets validate the effectiveness of the new approach.

Abstract

Self-supervised learning (SSL), which aims to learn meaningful prior representations from unlabeled data, has been proven effective for skeleton-based action understanding. Different from the image domain, skeleton data possesses sparser spatial structures and diverse representation forms, with the absence of background clues and the additional temporal dimension, presenting new challenges for spatial-temporal motion pretext task design. Recently, many endeavors have been made for skeleton-based SSL, achieving remarkable progress. However, a systematic and thorough review is still lacking. In this paper, we conduct, for the first time, a comprehensive survey on self-supervised skeleton-based action representation learning. Following the taxonomy of context-based, generative learning, and contrastive learning approaches, we make a thorough review and benchmark of existing works and shed light on the future possible directions. Remarkably, our investigation demonstrates that most SSL works rely on the single paradigm, learning representations of a single level, and are evaluated on the action recognition task solely, which leaves the generalization power of skeleton SSL models under-explored. To this end, a novel and effective SSL method for skeleton is further proposed, which integrates versatile representation learning objectives of different granularity, substantially boosting the generalization capacity for multiple skeleton downstream tasks. Extensive experiments under three large-scale datasets demonstrate our method achieves superior generalization performance on various downstream tasks, including recognition, retrieval, detection, and few-shot learning.