LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

TL;DR

This paper introduces HocSLM, a hierarchical skeleton-language model that enhances action recognition by integrating global-local spatio-temporal modeling with semantic-rich textual descriptions, achieving state-of-the-art results.

Contribution

The paper proposes a novel hierarchical global-local network combined with a large vision-language model and a sequential fusion module for improved semantic understanding in action recognition.

Findings

Achieves state-of-the-art performance on NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA datasets.

Enhances cross-modal semantic alignment between skeletal data and textual descriptions.

Improves modeling of complex spatio-temporal relationships in skeleton-based action recognition.

Abstract

Skeleton-based human action recognition has achieved remarkable progress in recent years. However, most existing GCN-based methods rely on short-range motion topologies, which not only struggle to capture long-range joint dependencies and complex temporal dynamics but also limit cross-modal semantic alignment and understanding due to insufficient modeling of action semantics. To address these challenges, we propose a hierarchical global-local skeleton-language model (HocSLM), enabling the large action model be more representative of action semantics. First, we design a hierarchical global-local network (HGLNet) that consists of a composite-topology spatial module and a dual-path hierarchical temporal module. By synergistically integrating multi-level global and local modules, HGLNet achieves dynamically collaborative modeling at both global and local scales while preserving prior knowledge of human physical structure, significantly enhancing the model's representation of complex spatio-temporal relationships. Then, a large vision-language model (VLM) is employed to generate textual descriptions by passing the original RGB video sequences to this model, providing the rich action semantics for further training the skeleton-language model. Furthermore, we introduce a skeleton-language sequential fusion module by combining the features from HGLNet and the generated descriptions, which utilizes a skeleton-language model (SLM) for aligning skeletal spatio-temporal features and textual action descriptions precisely within a unified semantic space. The SLM model could significantly enhance the HGLNet's semantic discrimination capabilities and cross-modal understanding abilities. Extensive experiments demonstrate that the proposed HocSLM achieves the state-of-the-art performance on three mainstream benchmark datasets: NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA.