Skeleton-Snippet Contrastive Learning with Multiscale Feature Fusion for Action Localization

TL;DR

This paper introduces a contrastive learning method with multiscale feature fusion for skeleton-based action localization, improving boundary detection and achieving state-of-the-art results.

Contribution

It proposes a snippet discrimination pretext task and a U-shaped feature fusion module to enhance skeleton-based action localization.

Findings

Improves action localization performance on BABEL dataset.

Achieves state-of-the-art transfer learning results on PKUMMD.

Enhances feature resolution for frame-level localization.

Abstract

The self-supervised pretraining paradigm has achieved great success in learning 3D action representations for skeleton-based action recognition using contrastive learning. However, learning effective representations for skeleton-based temporal action localization remains challenging and underexplored. Unlike video-level {action} recognition, detecting action boundaries requires temporally sensitive features that capture subtle differences between adjacent frames where labels change. To this end, we formulate a snippet discrimination pretext task for self-supervised pretraining, which densely projects skeleton sequences into non-overlapping segments and promotes features that distinguish them across videos via contrastive learning. Additionally, we build on strong backbones of skeleton-based action recognition models by fusing intermediate features with a U-shaped module to enhance feature resolution for frame-level localization. Our approach consistently improves existing skeleton-based contrastive learning methods for action localization on BABEL across diverse subsets and evaluation protocols. We also achieve state-of-the-art transfer learning performance on PKUMMD with pretraining on NTU RGB+D and BABEL.