HFGaussian: Learning Generalizable Gaussian Human with Integrated Human Features

TL;DR

HFGaussian introduces a real-time, generalizable Gaussian splatting method for 3D human representation that integrates biomechanical features like skeletons and key points from sparse images.

Contribution

The paper presents a novel approach combining Gaussian splatting with pose regression to estimate detailed 3D human features efficiently from limited input data.

Findings

Achieves real-time performance at 25 FPS.

Outperforms existing methods in 3D human reconstruction.

Effectively integrates biomechanical features into Gaussian representations.

Abstract

Recent advancements in radiance field rendering show promising results in 3D scene representation, where Gaussian splatting-based techniques emerge as state-of-the-art due to their quality and efficiency. Gaussian splatting is widely used for various applications, including 3D human representation. However, previous 3D Gaussian splatting methods either use parametric body models as additional information or fail to provide any underlying structure, like human biomechanical features, which are essential for different applications. In this paper, we present a novel approach called HFGaussian that can estimate novel views and human features, such as the 3D skeleton, 3D key points, and dense pose, from sparse input images in real time at 25 FPS. The proposed method leverages generalizable Gaussian splatting technique to represent the human subject and its associated features, enabling efficient and generalizable reconstruction. By incorporating a pose regression network and the feature splatting technique with Gaussian splatting, HFGaussian demonstrates improved capabilities over existing 3D human methods, showcasing the potential of 3D human representations with integrated biomechanics. We thoroughly evaluate our HFGaussian method against the latest state-of-the-art techniques in human Gaussian splatting and pose estimation, demonstrating its real-time, state-of-the-art performance.