Dual-stream Spatio-Temporal GCN-Transformer Network for 3D Human Pose Estimation

TL;DR

The paper introduces MixTGFormer, a dual-stream GCN-Transformer network that models both local and global spatio-temporal relationships for 3D human pose estimation, achieving state-of-the-art results.

Contribution

It proposes a novel dual-stream architecture combining GCN and Transformer with Mixformer blocks and SE layers for improved 3D pose estimation.

Findings

Achieved state-of-the-art P1 errors of 37.6mm on Human3.6M

Achieved state-of-the-art P1 errors of 15.7mm on MPI-INF-3DHP

Effectively fuses local skeletal and global features through dual streams.

Abstract

3D human pose estimation is a classic and important research direction in the field of computer vision. In recent years, Transformer-based methods have made significant progress in lifting 2D to 3D human pose estimation. However, these methods primarily focus on modeling global temporal and spatial relationships, neglecting local skeletal relationships and the information interaction between different channels. Therefore, we have proposed a novel method,the Dual-stream Spatio-temporal GCN-Transformer Network (MixTGFormer). This method models the spatial and temporal relationships of human skeletons simultaneously through two parallel channels, achieving effective fusion of global and local features. The core of MixTGFormer is composed of stacked Mixformers. Specifically, the Mixformer includes the Mixformer Block and the Squeeze-and-Excitation Layer ( SE Layer). It first extracts and fuses various information of human skeletons through two parallel Mixformer Blocks with different modes. Then, it further supplements the fused information through the SE Layer. The Mixformer Block integrates Graph Convolutional Networks (GCN) into the Transformer, enhancing both local and global information utilization. Additionally, we further implement its temporal and spatial forms to extract both spatial and temporal relationships. We extensively evaluated our model on two benchmark datasets (Human3.6M and MPI-INF-3DHP). The experimental results showed that, compared to other methods, our MixTGFormer achieved state-of-the-art results, with P1 errors of 37.6mm and 15.7mm on these datasets, respectively.