TransHuman: A Transformer-based Human Representation for Generalizable Neural Human Rendering

TL;DR

TransHuman introduces a transformer-based framework for neural human rendering that effectively models global relationships and handles dynamic motion, outperforming previous methods in generalization and efficiency.

Contribution

The paper proposes TransHuman, a novel transformer-based approach that learns human representations in canonical space and captures global part relationships, improving over SPC-based methods.

Findings

Achieves state-of-the-art performance on ZJU-MoCap and H36M datasets.

Effectively models global relationships between human parts.

Handles dynamic human motion with high efficiency.

Abstract

In this paper, we focus on the task of generalizable neural human rendering which trains conditional Neural Radiance Fields (NeRF) from multi-view videos of different characters. To handle the dynamic human motion, previous methods have primarily used a SparseConvNet (SPC)-based human representation to process the painted SMPL. However, such SPC-based representation i) optimizes under the volatile observation space which leads to the pose-misalignment between training and inference stages, and ii) lacks the global relationships among human parts that is critical for handling the incomplete painted SMPL. Tackling these issues, we present a brand-new framework named TransHuman, which learns the painted SMPL under the canonical space and captures the global relationships between human parts with transformers. Specifically, TransHuman is mainly composed of Transformer-based Human Encoding (TransHE), Deformable Partial Radiance Fields (DPaRF), and Fine-grained Detail Integration (FDI). TransHE first processes the painted SMPL under the canonical space via transformers for capturing the global relationships between human parts. Then, DPaRF binds each output token with a deformable radiance field for encoding the query point under the observation space. Finally, the FDI is employed to further integrate fine-grained information from reference images. Extensive experiments on ZJU-MoCap and H36M show that our TransHuman achieves a significantly new state-of-the-art performance with high efficiency. Project page: https://pansanity666.github.io/TransHuman/