SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers

TL;DR

SkelFormer is a new markerless 3D human pose and shape estimation method that uses skeletal transformers to improve robustness and generalization, especially under noisy and occluded conditions.

Contribution

The paper introduces a novel skeletal transformer architecture that separates 3D keypoint detection from pose estimation, enhancing accuracy and robustness in markerless motion capture.

Findings

Strong performance on multiple datasets

Robust to noise and occlusions

Ablation studies confirm module effectiveness

Abstract

We introduce SkelFormer, a novel markerless motion capture pipeline for multi-view human pose and shape estimation. Our method first uses off-the-shelf 2D keypoint estimators, pre-trained on large-scale in-the-wild data, to obtain 3D joint positions. Next, we design a regression-based inverse-kinematic skeletal transformer that maps the joint positions to pose and shape representations from heavily noisy observations. This module integrates prior knowledge about pose space and infers the full pose state at runtime. Separating the 3D keypoint detection and inverse-kinematic problems, along with the expressive representations learned by our skeletal transformer, enhance the generalization of our method to unseen noisy data. We evaluate our method on three public datasets in both in-distribution and out-of-distribution settings using three datasets, and observe strong performance with respect to prior works. Moreover, ablation experiments demonstrate the impact of each of the modules of our architecture. Finally, we study the performance of our method in dealing with noise and heavy occlusions and find considerable robustness with respect to other solutions.