Less is More: Decoder-Free Masked Modeling for Efficient Skeleton Representation Learning

TL;DR

SLiM introduces a decoder-free masked modeling framework for skeleton representation learning, combining contrastive learning with masked modeling, leading to state-of-the-art accuracy and significantly improved efficiency.

Contribution

It is the first to propose a decoder-free masked modeling approach for skeleton learning, integrating contrastive learning with masked modeling in a unified framework.

Findings

Achieves state-of-the-art performance on downstream tasks.

Reduces inference computational cost by 7.89x compared to existing MAE methods.

Effectively captures discriminative features without a decoder.

Abstract

The landscape of skeleton-based action representation learning has evolved from Contrastive Learning (CL) to Masked Auto-Encoder (MAE) architectures. However, each paradigm faces inherent limitations: CL often overlooks fine-grained local details, while MAE is burdened by computationally heavy decoders. Moreover, MAE suffers from severe computational asymmetry -- benefiting from efficient masking during pre-training but requiring exhaustive full-sequence processing for downstream tasks. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a novel unified framework that harmonizes masked modeling with contrastive learning via a shared encoder. By eschewing the reconstruction decoder, SLiM not only eliminates computational redundancy but also compels the encoder to capture discriminative features directly. SLiM is the first framework with decoder-free masked modeling of representative learning. Crucially, to prevent trivial reconstruction arising from high skeletal-temporal correlation, we introduce semantic tube masking, alongside skeletal-aware augmentations designed to ensure anatomical consistency across diverse temporal granularities. Extensive experiments demonstrate that SLiM consistently achieves state-of-the-art performance across all downstream protocols. Notably, our method delivers this superior accuracy with exceptional efficiency, reducing inference computational cost by 7.89x compared to existing MAE methods.