Semi-Supervised 3D Hand Shape and Pose Estimation with Label Propagation

TL;DR

This paper introduces a semi-supervised method using a Pose Alignment network to propagate 3D hand annotations from labeled to unlabeled frames, improving pose estimation accuracy in videos without extensive labeling.

Contribution

The novel Pose Alignment network enables effective propagation of 3D annotations in sparsely labeled videos, enhancing generalizability without fine-tuning.

Findings

Improved 3D hand pose estimation accuracy.

Effective annotation propagation on unseen videos.

No fine-tuning required for new videos.

Abstract

To obtain 3D annotations, we are restricted to controlled environments or synthetic datasets, leading us to 3D datasets with less generalizability to real-world scenarios. To tackle this issue in the context of semi-supervised 3D hand shape and pose estimation, we propose the Pose Alignment network to propagate 3D annotations from labelled frames to nearby unlabelled frames in sparsely annotated videos. We show that incorporating the alignment supervision on pairs of labelled-unlabelled frames allows us to improve the pose estimation accuracy. Besides, we show that the proposed Pose Alignment network can effectively propagate annotations on unseen sparsely labelled videos without fine-tuning.