SkeletonContext: Skeleton-side Context Prompt Learning for Zero-Shot Skeleton-based Action Recognition

TL;DR

SkeletonContext introduces a prompt-based framework that incorporates language-driven contextual semantics into skeleton motion representations, significantly improving zero-shot skeleton-based action recognition performance.

Contribution

It proposes a novel Cross-Modal Context Prompt Module and Key-Part Decoupling Module to enhance semantic grounding and robustness in recognizing unseen actions.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Effectively incorporates contextual cues to distinguish similar actions.

Improves zero-shot recognition accuracy in skeleton-based action recognition.

Abstract

Zero-shot skeleton-based action recognition aims to recognize unseen actions by transferring knowledge from seen categories through semantic descriptions. Most existing methods typically align skeleton features with textual embeddings within a shared latent space. However, the absence of contextual cues, such as objects involved in the action, introduces an inherent gap between skeleton and semantic representations, making it difficult to distinguish visually similar actions. To address this, we propose SkeletonContext, a prompt-based framework that enriches skeletal motion representations with language-driven contextual semantics. Specifically, we introduce a Cross-Modal Context Prompt Module, which leverages a pretrained language model to reconstruct masked contextual prompts under guidance derived from LLMs. This design effectively transfers linguistic context to the skeleton encoder for instance-level semantic grounding and improved cross-modal alignment. In addition, a Key-Part Decoupling Module is incorporated to decouple motion-relevant joint features, ensuring robust action understanding even in the absence of explicit object interactions. Extensive experiments on multiple benchmarks demonstrate that SkeletonContext achieves state-of-the-art performance under both conventional and generalized zero-shot settings, validating its effectiveness in reasoning about context and distinguishing fine-grained, visually similar actions.