GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers

TL;DR

This paper introduces a method combining 3D Gaussian Splatting with transformer-based adjustments to reconstruct detailed 3D human models from a single image, suitable for real-time sports applications without needing 3D ground truth.

Contribution

It presents a novel approach that uses SMPL mesh vertices as initial Gaussian positions and trains a transformer to predict adjustments, enabling accurate 3D human reconstruction from monocular images.

Findings

Achieves near real-time 3D human reconstruction from a single image.

Does not require 3D ground truth or expensive diffusion models.

Improves pose estimation by incorporating rendering as an auxiliary task.

Abstract

Reconstructing posed 3D human models from monocular images has important applications in the sports industry, including performance tracking, injury prevention and virtual training. In this work, we combine 3D human pose and shape estimation with 3D Gaussian Splatting (3DGS), a representation of the scene composed of a mixture of Gaussians. This allows training or fine-tuning a human model predictor on multi-view images alone, without 3D ground truth. Predicting such mixtures for a human from a single input image is challenging due to self-occlusions and dependence on articulations, while also needing to retain enough flexibility to accommodate a variety of clothes and poses. Our key observation is that the vertices of standardized human meshes (such as SMPL) can provide an adequate spatial density and approximate initial position for the Gaussians. We can then train a transformer model to jointly predict comparatively small adjustments to these positions, as well as the other 3DGS attributes and the SMPL parameters. We show empirically that this combination (using only multi-view supervision) can achieve near real-time inference of 3D human models from a single image without expensive diffusion models or 3D points supervision, thus making it ideal for the sport industry at any level. More importantly, rendering is an effective auxiliary objective to refine 3D pose estimation by accounting for clothes and other geometric variations. The code is available at https://github.com/prosperolo/GST.