Bootstrapped Representation Learning for Skeleton-Based Action Recognition

TL;DR

This paper introduces a self-supervised learning approach for skeleton-based action recognition, extending BYOL with novel data augmentation and multi-viewpoint sampling, achieving state-of-the-art results on NTU datasets.

Contribution

It proposes a new self-supervised learning framework for skeleton data, incorporating asymmetric augmentations and multi-viewpoint sampling, with improvements via knowledge distillation.

Findings

Outperforms existing methods on NTU-60 and NTU-120 datasets.

Effective semi-supervised learning with knowledge distillation.

Consistent improvements in linear and semi-supervised evaluations.

Abstract

In this work, we study self-supervised representation learning for 3D skeleton-based action recognition. We extend Bootstrap Your Own Latent (BYOL) for representation learning on skeleton sequence data and propose a new data augmentation strategy including two asymmetric transformation pipelines. We also introduce a multi-viewpoint sampling method that leverages multiple viewing angles of the same action captured by different cameras. In the semi-supervised setting, we show that the performance can be further improved by knowledge distillation from wider networks, leveraging once more the unlabeled samples. We conduct extensive experiments on the NTU-60 and NTU-120 datasets to demonstrate the performance of our proposed method. Our method consistently outperforms the current state of the art on both linear evaluation and semi-supervised benchmarks.