Generalizable Human Gaussian Splatting via Multi-view Semantic Consistency

TL;DR

This paper introduces a novel approach for human Gaussian splatting that leverages multi-view semantic consistency to improve 3D Gaussian localization and rendering quality from sparse views.

Contribution

It proposes a new method using cross-view attention to unproject and recalibrate latent embeddings, addressing feature inconsistency across views.

Findings

Improves 3D Gaussian localization accuracy.

Enhances human rendering quality from sparse views.

Outperforms existing methods on benchmark datasets.

Abstract

Recently, generalizable human Gaussian splatting from sparse-view inputs has been actively studied for the photorealistic human rendering. Most existing methods rely on explicit geometric constraints or predefined structural representations to accurately position 3D Gaussians. Although these approaches have shown the remarkable progress in this field, they still suffer from inconsistent feature representations across multi-view inputs due to complex articulations of the human body and limited overlaps between different views. To address this problem, we propose a novel method to accurately localize 3D Gaussians and ultimately improve the quality of human rendering. The key idea is to unproject latent embeddings encoded from each viewpoint into a shared 3D space through predicted depth maps and recalibrate them belonging to the same body part based on cross-view attention. This helps the model resolve the spatial ambiguity occurring in highly textured regions as well as occluded body parts, thus leading to the accurate localization of 3D Gaussians. Experimental results on benchmark datasets show that the proposed method efficiently improves the performance of generalizable human Gaussian splatting from sparse-view inputs.