HyLiFormer: Hyperbolic Linear Attention for Skeleton-based Human Action Recognition

TL;DR

HyLiFormer introduces a hyperbolic linear attention Transformer that efficiently models hierarchical skeleton data, reducing computational costs while maintaining high accuracy in human action recognition.

Contribution

It presents a novel hyperbolic linear attention mechanism with a Hyperbolic Transformation module, enhancing efficiency and hierarchical modeling in skeleton-based action recognition.

Findings

Significantly reduces computational complexity

Maintains high accuracy on NTU datasets

Effective hierarchical structure modeling

Abstract

Transformers have demonstrated remarkable performance in skeleton-based human action recognition, yet their quadratic computational complexity remains a bottleneck for real-world applications. To mitigate this, linear attention mechanisms have been explored but struggle to capture the hierarchical structure of skeleton data. Meanwhile, the Poincar\'e model, as a typical hyperbolic geometry, offers a powerful framework for modeling hierarchical structures but lacks well-defined operations for existing mainstream linear attention. In this paper, we propose HyLiFormer, a novel hyperbolic linear attention Transformer tailored for skeleton-based action recognition. Our approach incorporates a Hyperbolic Transformation with Curvatures (HTC) module to map skeleton data into hyperbolic space and a Hyperbolic Linear Attention (HLA) module for efficient long-range dependency modeling. Theoretical analysis and extensive experiments on NTU RGB+D and NTU RGB+D 120 datasets demonstrate that HyLiFormer significantly reduces computational complexity while preserving model accuracy, making it a promising solution for efficiency-critical applications.