Learning Sequential Contexts using Transformer for 3D Hand Pose Estimation

TL;DR

This paper introduces SeTHPose, a transformer-based method for 3D hand pose estimation that leverages sequential visual data to improve accuracy, achieving state-of-the-art results on multiple datasets.

Contribution

The paper presents a novel sequential learning approach using transformers combined with graph neural networks for improved 3D hand pose estimation.

Findings

SeTHPose outperforms existing methods on STB and MuViHand datasets.

The method effectively captures temporal and angular context for hand pose estimation.

Achieves new state-of-the-art accuracy in 3D hand pose estimation.

Abstract

3D hand pose estimation (HPE) is the process of locating the joints of the hand in 3D from any visual input. HPE has recently received an increased amount of attention due to its key role in a variety of human-computer interaction applications. Recent HPE methods have demonstrated the advantages of employing videos or multi-view images, allowing for more robust HPE systems. Accordingly, in this study, we propose a new method to perform Sequential learning with Transformer for Hand Pose (SeTHPose) estimation. Our SeTHPose pipeline begins by extracting visual embeddings from individual hand images. We then use a transformer encoder to learn the sequential context along time or viewing angles and generate accurate 2D hand joint locations. Then, a graph convolutional neural network with a U-Net configuration is used to convert the 2D hand joint locations to 3D poses. Our experiments show that SeTHPose performs well on both hand sequence varieties, temporal and angular. Also, SeTHPose outperforms other methods in the field to achieve new state-of-the-art results on two public available sequential datasets, STB and MuViHand.