Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentation

TL;DR

This paper introduces an unsupervised method for skeleton-based temporal action segmentation using a sequence-to-sequence autoencoder and motion words, outperforming existing methods on multiple datasets.

Contribution

It presents a novel unsupervised approach that leverages skeleton motion words and a disentangled embedding space for effective action segmentation.

Findings

Outperforms current state-of-the-art unsupervised methods

Effective on multiple skeleton datasets

Utilizes a novel skeleton motion words concept

Abstract

Current state-of-the-art methods for skeleton-based temporal action segmentation are predominantly supervised and require annotated data, which is expensive to collect. In contrast, existing unsupervised temporal action segmentation methods have focused primarily on video data, while skeleton sequences remain underexplored, despite their relevance to real-world applications, robustness, and privacy-preserving nature. In this paper, we propose a novel approach for unsupervised skeleton-based temporal action segmentation. Our method utilizes a sequence-to-sequence temporal autoencoder that keeps the information of the different joints disentangled in the embedding space. Latent skeleton sequences are then divided into non-overlapping patches and quantized to obtain distinctive skeleton motion words, driving the discovery of semantically meaningful action clusters. We thoroughly evaluate the proposed approach on three widely used skeleton-based datasets, namely HuGaDB, LARa, and BABEL. The results demonstrate that our model outperforms the current state-of-the-art unsupervised temporal action segmentation methods. Code is available at https://github.com/bachlab/SMQ .