USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation

TL;DR

USDRL introduces a novel skeleton-based dense representation learning framework that employs multi-grained feature decorrelation and a dense spatio-temporal encoder, significantly improving performance on various action recognition and detection benchmarks.

Contribution

The paper proposes a unified dense representation learning method with feature decorrelation and a specialized encoder, addressing limitations of negative-based contrastive learning and enhancing dense prediction capabilities.

Findings

Outperforms state-of-the-art on NTU-60, NTU-120, PKU-MMD I, and PKU-MMD II datasets.

Effectively captures fine-grained action features for dense prediction tasks.

Demonstrates significant improvements in action recognition, retrieval, and detection.

Abstract

Contrastive learning has achieved great success in skeleton-based representation learning recently. However, the prevailing methods are predominantly negative-based, necessitating additional momentum encoder and memory bank to get negative samples, which increases the difficulty of model training. Furthermore, these methods primarily concentrate on learning a global representation for recognition and retrieval tasks, while overlooking the rich and detailed local representations that are crucial for dense prediction tasks. To alleviate these issues, we introduce a Unified Skeleton-based Dense Representation Learning framework based on feature decorrelation, called USDRL, which employs feature decorrelation across temporal, spatial, and instance domains in a multi-grained manner to reduce redundancy among dimensions of the representations to maximize information extraction from features. Additionally, we design a Dense Spatio-Temporal Encoder (DSTE) to capture fine-grained action representations effectively, thereby enhancing the performance of dense prediction tasks. Comprehensive experiments, conducted on the benchmarks NTU-60, NTU-120, PKU-MMD I, and PKU-MMD II, across diverse downstream tasks including action recognition, action retrieval, and action detection, conclusively demonstrate that our approach significantly outperforms the current state-of-the-art (SOTA) approaches. Our code and models are available at https://github.com/wengwanjiang/USDRL.