Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition

TL;DR

This paper introduces Frequency-Aware Diffusion for Skeleton-Text Matching (FDSM), a novel method that enhances zero-shot skeleton action recognition by recovering fine-grained motion details and addressing spectral bias in diffusion models.

Contribution

It proposes a spectral residual module, adaptive spectral loss, and curriculum-based semantic abstraction to improve zero-shot skeleton action recognition performance.

Findings

Achieves state-of-the-art results on NTU RGB+D, PKU-MMD, and Kinetics-skeleton datasets.

Effectively recovers high-frequency motion details.

Addresses spectral bias in diffusion models.

Abstract

Human action recognition is pivotal in computer vision, with applications ranging from surveillance to human-robot interaction. Despite the effectiveness of supervised skeleton-based methods, their reliance on exhaustive annotation limits generalization to novel actions. Zero-Shot Skeleton Action Recognition (ZSAR) emerges as a promising paradigm, yet it faces challenges due to the spectral bias of diffusion models, which oversmooth high-frequency dynamics. Here, we propose Frequency-Aware Diffusion for Skeleton-Text Matching (FDSM), integrating a Semantic-Guided Spectral Residual Module, a Timestep-Adaptive Spectral Loss, and Curriculum-based Semantic Abstraction to address these challenges. Our approach effectively recovers fine-grained motion details, achieving state-of-the-art performance on NTU RGB+D, PKU-MMD, and Kinetics-skeleton datasets. Code has been made available at https://github.com/yuzhi535/FDSM. Project homepage: https://yuzhi535.github.io/FDSM.github.io/