An Information Compensation Framework for Zero-Shot Skeleton-based Action Recognition

TL;DR

This paper proposes an information compensation framework utilizing multi-granularity semantic interaction and alignment to enhance zero-shot skeleton-based action recognition, achieving significant improvements on multiple benchmarks.

Contribution

It introduces a novel multi-level alignment approach and a new loss sampling method to improve semantic feature richness and classification accuracy in zero-shot action recognition.

Findings

Achieves state-of-the-art performance on NTU RGB+D, NTU RGB+D 120, and PKU-MMD datasets.

Demonstrates that multi-granularity semantic features improve action cluster differentiation.

Validates the effectiveness of the proposed information compensation framework.

Abstract

Zero-shot human skeleton-based action recognition aims to construct a model that can recognize actions outside the categories seen during training. Previous research has focused on aligning sequences' visual and semantic spatial distributions. However, these methods extract semantic features simply. They ignore that proper prompt design for rich and fine-grained action cues can provide robust representation space clustering. In order to alleviate the problem of insufficient information available for skeleton sequences, we design an information compensation learning framework from an information-theoretic perspective to improve zero-shot action recognition accuracy with a multi-granularity semantic interaction mechanism. Inspired by ensemble learning, we propose a multi-level alignment (MLA) approach to compensate information for action classes. MLA aligns multi-granularity embeddings with visual embedding through a multi-head scoring mechanism to distinguish semantically similar action names and visually similar actions. Furthermore, we introduce a new loss function sampling method to obtain a tight and robust representation. Finally, these multi-granularity semantic embeddings are synthesized to form a proper decision surface for classification. Significant action recognition performance is achieved when evaluated on the challenging NTU RGB+D, NTU RGB+D 120, and PKU-MMD benchmarks and validate that multi-granularity semantic features facilitate the differentiation of action clusters with similar visual features.