SCALE: Semantic- and Confidence-Aware Conditional Variational Autoencoder for Zero-shot Skeleton-based Action Recognition

TL;DR

SCALE introduces a novel energy-based framework using a conditional VAE for zero-shot skeleton action recognition, leveraging text semantics and confidence measures to improve class separation without generating samples.

Contribution

The paper proposes SCALE, a deterministic, text-conditioned VAE with a listwise energy loss and latent prototypes, enhancing zero-shot skeleton action recognition beyond prior methods.

Findings

SCALE outperforms previous VAE and alignment-based methods on NTU datasets.

The approach effectively separates semantically similar classes without sample generation.

Incorporating posterior uncertainty improves decision margins and handling ambiguous instances.

Abstract

Zero-shot skeleton-based action recognition (ZSAR) aims to recognize action classes without any training skeletons from those classes, relying instead on auxiliary semantics from text. Existing approaches frequently depend on explicit skeleton-text alignment, which can be brittle when action names underspecify fine-grained dynamics and when unseen classes are semantically confusable. We propose SCALE, a lightweight and deterministic Semantic- and Confidence-Aware Listwise Energy-based framework that formulates ZSAR as class-conditional energy ranking. SCALE builds a text-conditioned Conditional Variational Autoencoder where frozen text representations parameterize both the latent prior and the decoder, enabling likelihood-based evaluation for unseen classes without generating samples at test time. To separate competing hypotheses, we introduce a semantic- and confidence-aware listwise energy loss that emphasizes semantically similar hard negatives and incorporates posterior uncertainty to adapt decision margins and reweight ambiguous training instances. Additionally, we utilize a latent prototype contrast objective to align posterior means with text-derived latent prototypes, improving semantic organization and class separability without direct feature matching. Experiments on NTU-60 and NTU-120 datasets show that SCALE consistently improves over prior VAE- and alignment-based baselines while remaining competitive with diffusion-based methods.