Skeleton Transformer Networks: 3D Human Pose and Skinned Mesh from Single RGB Image

TL;DR

SkeletonNet is an end-to-end framework that predicts 3D joint positions and bone rotations from a single RGB image, enabling realistic skinned mesh animations and advancing 3D human pose estimation.

Contribution

It introduces a novel two-step regression approach for joint position and bone rotation prediction from a single image, utilizing a new 3D pose dataset for training.

Findings

Accurately predicts 3D human pose and bone rotations from single images.

Generates realistic skinned mesh animations.

Outperforms existing methods in in-the-wild scenarios.

Abstract

In this paper, we present Skeleton Transformer Networks (SkeletonNet), an end-to-end framework that can predict not only 3D joint positions but also 3D angular pose (bone rotations) of a human skeleton from a single color image. This in turn allows us to generate skinned mesh animations. Here, we propose a two-step regression approach. The first step regresses bone rotations in order to obtain an initial solution by considering skeleton structure. The second step performs refinement based on heatmap regressor using a 3D pose representation called cross heatmap which stacks heatmaps of xy and zy coordinates. By training the network using the proposed 3D human pose dataset that is comprised of images annotated with 3D skeletal angular poses, we showed that SkeletonNet can predict a full 3D human pose (joint positions and bone rotations) from a single image in-the-wild.