Learning Efficient and Generalizable Human Representation with Human Gaussian Model

TL;DR

This paper introduces the Human Gaussian Graph, a novel framework that models relationships between 3D Gaussians and human mesh vertices, enabling efficient and generalizable human avatar reconstruction from videos.

Contribution

The paper proposes the Human Gaussian Graph with intra- and inter-node operations to leverage information across frames for improved human avatar modeling.

Findings

Effective in novel view synthesis

Supports pose animation

Outperforms previous methods in efficiency and generalization

Abstract

Modeling animatable human avatars from videos is a long-standing and challenging problem. While conventional methods require per-instance optimization, recent feed-forward methods have been proposed to generate 3D Gaussians with a learnable network. However, these methods predict Gaussians for each frame independently, without fully capturing the relations of Gaussians from different timestamps. To address this, we propose Human Gaussian Graph to model the connection between predicted Gaussians and human SMPL mesh, so that we can leverage information from all frames to recover an animatable human representation. Specifically, the Human Gaussian Graph contains dual layers where Gaussians are the first layer nodes and mesh vertices serve as the second layer nodes. Based on this structure, we further propose the intra-node operation to aggregate various Gaussians connected to one mesh vertex, and inter-node operation to support message passing among mesh node neighbors. Experimental results on novel view synthesis and novel pose animation demonstrate the efficiency and generalization of our method.