SpATr: MoCap 3D Human Action Recognition based on Spiral Auto-encoder and Transformer Network

TL;DR

SpATr introduces a novel 3D human action recognition method using spiral auto-encoders and transformers, effectively capturing spatial and temporal features from fixed-topology mesh sequences with efficient memory use.

Contribution

The paper presents a new approach for 3D human action recognition that combines spiral convolutions with a transformer, specifically designed for fixed-topology mesh data, improving scalability and memory efficiency.

Findings

Competitive performance on Babel, MoVi, and BMLrub datasets

Efficient memory usage compared to prior methods

Effective long-range temporal dependency modeling

Abstract

Recent technological advancements have significantly expanded the potential of human action recognition through harnessing the power of 3D data. This data provides a richer understanding of actions, including depth information that enables more accurate analysis of spatial and temporal characteristics. In this context, We study the challenge of 3D human action recognition.Unlike prior methods, that rely on sampling 2D depth images, skeleton points, or point clouds, often leading to substantial memory requirements and the ability to handle only short sequences, we introduce a novel approach for 3D human action recognition, denoted as SpATr (Spiral Auto-encoder and Transformer Network), specifically designed for fixed-topology mesh sequences. The SpATr model disentangles space and time in the mesh sequences. A lightweight auto-encoder, based on spiral convolutions, is employed to extract spatial geometrical features from each 3D mesh. These convolutions are lightweight and specifically designed for fix-topology mesh data. Subsequently, a temporal transformer, based on self-attention, captures the temporal context within the feature sequence. The self-attention mechanism enables long-range dependencies capturing and parallel processing, ensuring scalability for long sequences. The proposed method is evaluated on three prominent 3D human action datasets: Babel, MoVi, and BMLrub, from the Archive of Motion Capture As Surface Shapes (AMASS). Our results analysis demonstrates the competitive performance of our SpATr model in 3D human action recognition while maintaining efficient memory usage. The code and the training results will soon be made publicly available at https://github.com/h-bouzid/spatr.