Point-Supervised Skeleton-Based Human Action Segmentation

TL;DR

This paper introduces a point-supervised framework for skeleton-based human action segmentation that requires only one labeled frame per action, leveraging multimodal data and pseudo-labeling techniques to achieve competitive results with less annotation effort.

Contribution

The paper proposes a novel point-supervised approach using multimodal skeleton data and a prototype similarity method, reducing annotation costs while maintaining high segmentation performance.

Findings

Achieves competitive performance with fewer annotations.

Outperforms some fully-supervised methods on benchmark datasets.

Establishes new benchmarks for point-supervised segmentation.

Abstract

Skeleton-based temporal action segmentation is a fundamental yet challenging task, playing a crucial role in enabling intelligent systems to perceive and respond to human activities. While fully-supervised methods achieve satisfactory performance, they require costly frame-level annotations and are sensitive to ambiguous action boundaries. To address these issues, we introduce a point-supervised framework for skeleton-based action segmentation, where only a single frame per action segment is labeled. We leverage multimodal skeleton data, including joint, bone, and motion information, encoded via a pretrained unified model to extract rich feature representations. To generate reliable pseudo-labels, we propose a novel prototype similarity method and integrate it with two existing methods: energy function and constrained K-Medoids clustering. Multimodal pseudo-label integration is proposed to enhance the reliability of the pseudo-label and guide the model training. We establish new benchmarks on PKU-MMD (X-Sub and X-View), MCFS-22, and MCFS-130, and implement baselines for point-supervised skeleton-based human action segmentation. Extensive experiments show that our method achieves competitive performance, even surpassing some fully-supervised methods while significantly reducing annotation effort.