DevilSight: Augmenting Monocular Human Avatar Reconstruction through a Virtual Perspective

TL;DR

DevilSight introduces a framework that enhances monocular human avatar reconstruction by leveraging a generative model for multi-view supervision, resulting in more detailed and plausible 3D avatars from single videos.

Contribution

The paper proposes using Human4DiT for multi-view supervision and introduces strategies for consistent motion and high-resolution detail enhancement in avatar reconstruction.

Findings

Outperforms recent state-of-the-art methods

Enriches details in unseen regions of avatars

Improves regularization and artifact mitigation

Abstract

We present a novel framework to reconstruct human avatars from monocular videos. Recent approaches have struggled either to capture the fine-grained dynamic details from the input or to generate plausible details at novel viewpoints, which mainly stem from the limited representational capacity of the avatar model and insufficient observational data. To overcome these challenges, we propose to leverage the advanced video generative model, Human4DiT, to generate the human motions from alternative perspective as an additional supervision signal. This approach not only enriches the details in previously unseen regions but also effectively regularizes the avatar representation to mitigate artifacts. Furthermore, we introduce two complementary strategies to enhance video generation: To ensure consistent reproduction of human motion, we inject the physical identity into the model through video fine-tuning. For higher-resolution outputs with finer details, a patch-based denoising algorithm is employed. Experimental results demonstrate that our method outperforms recent state-of-the-art approaches and validate the effectiveness of our proposed strategies.

Tables

Table 1. Table 1 : Quantitative comparisons on input view reconstruction and novel view synthesis with HumanNeRF  [ 75 ] , GaussianAvatar  [ 16 ] and AnimatableGaussians (AG for short)  [ 33 ] . The best and the second best are highlighted in bold and underlined fonts, respectively.

Dataset	Method	Input View			Novel View 1			Novel View 2
		PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$
THuman4.0	HumanNeRF	$30.69$	$0.9515$	$0.0841$	$25.84$	$0.9438$	$\underset{¯}{0.1051}$	$24.97$	$0.9372$	$\underset{¯}{0.1163}$
	GaussianAvatar	$31.13$	$0.9709$	$0.0820$	$22.25$	$0.9533$	$0.1271$	$22.02$	$0.9500$	$0.1346$
	AG	$34.06$	$0.9798$	$0.0554$	$\underset{¯}{26.55}$	$\underset{¯}{0.9588}$	$0.1071$	$\underset{¯}{25.65}$	$\underset{¯}{0.9542}$	$0.1193$
	Ours	$\underset{¯}{32.97}$	$\underset{¯}{0.9769}$	$\underset{¯}{0.0558}$	$26.79$	$0.9605$	$0.0995$	$25.98$	$0.9560$	$0.1111$
Mono2K	HumanNeRF	$27.91$	$0.9315$	$0.0822$	$26.02$	$0.9305$	$0.0873$	$25.45$	$0.9263$	$\underset{¯}{0.0941}$
	GaussianAvatar	$29.13$	$0.9729$	$0.0734$	$21.26$	$0.9549$	$0.1071$	$20.83$	$0.9542$	$0.1121$
	AG	$33.15$	$0.9807$	$0.0533$	$26.29$	$\underset{¯}{0.9654}$	$\underset{¯}{0.0819}$	$\underset{¯}{25.60}$	$\underset{¯}{0.9591}$	$0.0972$
	Ours	$\underset{¯}{32.19}$	$\underset{¯}{0.9768}$	$\underset{¯}{0.0614}$	$\underset{¯}{26.26}$	$0.9670$	$0.0788$	$25.73$	$0.9633$	$0.0898$

Table 2. Table A : Quantitative comparisons with AnimatableGaussians (AG for short)  [ 33 ] and the generated video (Gen. for short ) from finetuned Human4DiT on THuman4.0. The best and the second best are highlighted in bold and underlined fonts, respectively.

View	Method	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	FID$\downarrow$
Input	AG	$34.06$	$0.9798$	$0.0554$	$17.91$
	Ours	$\underset{¯}{32.97}$	$\underset{¯}{0.9769}$	$\underset{¯}{0.0558}$	$\underset{¯}{19.16}$
	Gen.	$32.38$	$0.9697$	$0.0808$	$38.55$
Novel	AG	$26.10$	$0.9565$	$0.1132$	100.34
	Ours	$26.39$	$\underset{¯}{0.9583}$	$\underset{¯}{0.1053}$	$\underset{¯}{71.06}$
	Gen.	$\underset{¯}{26.18}$	$0.9598$	$0.1017$	$59.25$

Table 3. Table B : Quantitative comparisons on input view reconstruction and novel view synthesis with HumanNeRF  [ 75 ] , GaussianAvatar  [ 16 ] and AnimatableGaussians (AG for short)  [ 33 ] . The best result is highlighted in bold fonts.

Dataset	Method	Input View			Novel View 1			Novel View 2
		PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$
THuman4.0 subject00	HumanNeRF	$30.84$	$0.9597$	$0.0940$	$27.01$	$0.9628$	$0.1181$	$26.41$	$0.9586$	$0.1229$
	GaussianAvatar	$31.08$	$0.9744$	$0.0777$	$23.10$	$0.9668$	$0.1317$	$22.94$	$0.9641$	$0.1345$
	AG	$32.76$	$0.9820$	$0.0535$	$28.43$	$0.9720$	$0.1047$	$27.83$	$0.9690$	$0.1130$
	Ours	$32.32$	$0.9801$	$0.0532$	$28.51$	$0.9738$	$0.0959$	$28.08$	$0.9710$	$0.1028$
THuman4.0 subject01	HumanNeRF	$30.22$	$0.9393$	$0.0883$	$26.27$	$0.9386$	$0.0994$	$24.91$	$0.9186$	$0.1168$
	GaussianAvatar	$30.91$	$0.9652$	$0.0906$	$22.50$	$0.9500$	$0.1286$	$21.70$	$0.9373$	$0.1398$
	AG	$34.19$	$0.9749$	$0.0623$	$26.23$	$0.9540$	$0.1157$	$24.41$	$0.9388$	$0.1360$
	Ours	$32.77$	$0.9749$	$0.0609$	$26.39$	$0.9573$	$0.1032$	$24.68$	$0.9418$	$0.1243$
THuman4.0 subject02	HumanNeRF	$31.00$	$0.9556$	$0.0699$	$24.25$	$0.9301$	$0.0977$	$23.58$	$0.9344$	$0.1091$
	GaussianAvatar	$31.39$	$0.9731$	$0.0777$	$21.16$	$0.9430$	$0.1211$	$21.41$	$0.9485$	$0.1295$
	AG	$35.24$	$0.9824$	$0.0503$	$25.00$	$0.9505$	$0.1010$	$24.71$	$0.9549$	$0.1088$
	Ours	$33.82$	$0.9788$	$0.0533$	$25.46$	$0.9505$	$0.0995$	$25.18$	$0.9552$	$0.1061$
Mono2K male	HumanNeRF	$26.90$	$0.9206$	$0.0836$	$25.21$	$0.9198$	$0.0879$	$25.27$	$0.9244$	$0.0948$
	GaussianAvatar	$28.35$	$0.9693$	$0.0758$	$19.95$	$0.9473$	$0.1084$	$20.07$	$0.9506$	$0.1144$
	AG	$32.44$	$0.9774$	$0.0566$	$25.40$	$0.9601$	$0.0841$	$25.50$	$0.9561$	$0.0987$
	Ours	$31.45$	$0.9733$	$0.0656$	$25.28$	$0.9627$	$0.0786$	$25.53$	$0.9623$	$0.0892$
Mono2K female	HumanNeRF	$28.91$	$0.9424$	$0.0808$	$26.83$	$0.9412$	$0.0866$	$25.62$	$0.9282$	$0.0933$
	GaussianAvatar	$29.91$	$0.9764$	$0.0709$	$22.57$	$0.9625$	$0.1058$	$21.58$	$0.9578$	$0.1098$
	AG	$33.86$	$0.9839$	$0.0500$	$27.18$	$0.9706$	$0.0796$	$25.70$	$0.9621$	$0.0957$
	Ours	$32.94$	$0.9803$	$0.0572$	$27.24$	$0.9713$	$0.0790$	$25.94$	$0.9643$	$0.0904$

Equations

$$f(\mathbf{x}|\mathbf{\mu },\mathbf{\Sigma })\propto \exp \left(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^{\top }{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu })\right),$$

$${\mathcal{G}}_f(\mathbf{\Theta }),{\mathcal{G}}_b(\mathbf{\Theta })\leftarrow \mathcal{U}({\mathcal{P}}_f(\mathbf{\Theta }),{\mathcal{P}}_b(\mathbf{\Theta })),$$

$${\mathcal{L}}_{\mathrm{diffusion}}={\mathbb{E}}_{{\mathbf{z}}_t,c_{\mathrm{ref}},c_{\Theta },ϵ,t}\left({\Vert ϵ-{ϵ}_{\theta }\left({\mathbf{z}}_t,c_{\mathrm{ref}},c_{\Theta },t\right)\Vert }_2^2\right)$$

$$\mathcal{L}={\mathcal{L}}_{\mathrm{rgb}}+{\lambda }_{\mathrm{LPIPS}}{\mathcal{L}}_{\mathrm{LPIPS}}+{\lambda }_{\mathrm{offset}}{\mathcal{L}}_{\mathrm{offset}}+{\lambda }_{\mathrm{scale}}{\mathcal{L}}_{\mathrm{scale}},$$

$${\mathcal{L}}_{\mathrm{offset}}=\frac{1}{N}\sum _{n=1}^N\Vert \Delta {\mathbf{\mu }}_n{\Vert }_2,$$

$${\mathcal{L}}_{\mathrm{scale}}=\frac{1}{3}\sum _{c\in \{x,y,z\}}\left|{\mathbf{s}}_c-\frac{1}{3}\sum _{c\in \{x,y,z\}}{\mathbf{s}}_c\right|,$$

Full text

1 Introduction

Reconstructing high-fidelity and dynamically detailed human avatars holds significant value across a wide range of applications, including gaming, VR/AR technologies, and the movie industry. However, despite notable advancements in human avatar reconstruction methods [36, 49, 65, 32, 90, 91, 33, 6] utilizing multi-view inputs, it still remains a challenge to create an avatar with high-realistic texture details and dynamic fidelity from monocular video.

Previous approaches represent the human avatar with respect to deformable human templates [40, 46], in terms of structured latent codes [50] or implicit 3D representations [75, 17, 16, 10, 25, 14]. By constructing avatars in a canonical pose, these approaches align multi-frame observations into a unified 3D space and leverage pose-conditioned neural networks to interpolate dynamic details through spatial smoothness priors. While this framework ensures multi-view consistency, it struggles to recover high-frequency details due to limited cross-view correlation in the input frames and the required fusion operation. Furthermore, the strong reliance on the naked body template limits its applicability in scenarios involving loose clothing.

Recently, generative models [57, 7, 76] show the potential to “imagine” previously unseen content from text or images. Moreover, some works [37, 63, 39] achieve the generation and reconstruction of 3D objects by adding camera control. Based on these generative models, many methods [28, 22, 41, 2, 23, 13] leverage generative priors to enhance 3D reconstruction by score distillation sampling (SDS) [52] or explicitly generating multi-view images as pseudo ground truths. While these methods have achieved high-quality 3D static scene reconstruction from sparse-view inputs, reconstructing dynamic sequences remains a significant challenge. This difficulty primarily arises from generating video while maintaining the identity consistency across time and viewpoints. In this paper, we propose leveraging Human4DiT [59] to generate an identical motion sequence from an alternative perspective, thereby providing supplementary supervision signals for reconstruction. Recent advancements have demonstrated the impressive capability of diffusion transformers (DiT) [47] to incorporate control signals from various dimensions. Specifically designed to produce spatio-temporally coherent human videos, Human4DiT supplies the view and temporal consistency priors essential to our approach. In practice, we choose to generate only one video from the back view, for its efficiency and effectiveness to address the majority of unseen regions.

Nevertheless, directly generating the back-view video using Human4DiT presents challenges in accurately reproducing human motions that are physically consistent with the input video. This limitation arises because the generation is conditioned on an identity embedding extracted from a single reference image, which is insufficient to encapsulate the dynamic characteristics inherent to human identity. Consequently, generative models pretrained exclusively on such embeddings also struggle to accurately reproduce the motion. Drawing inspiration from DreamBooth [58], we propose Physical Identity Inversion through model finetuning, enabling video generation that aligns with the physical motion present in the input video. To prevent overfitting and maintain view consistency priors, we have meticulously designed our fine-tuning strategy and carefully selected the parameters involved. Moreover, in order to bridge the resolution gap between captured and generated videos and to mitigate potential artifacts arising from this difference [83], we have developed an innovative algorithm that effectively doubles the resolution of generated videos. Given that DiT-based models are constrained to denoising at fixed resolution, the algorithm involves generating videos by partitioning them into patches. Benefiting from these technical designs, our method achieves superior reconstruction quality compared to previous state-of-the-art methods.

As shown in the experiments, our method achieves high-fidelity human reconstruction with dynamic details and supports dynamically consistent image synthesis at novel views. In summary, our contributions are:

• •

We propose a novel framework for high-fidelity dynamic human-centric reconstruction from monocular video, which leverage the capability of video generative model to complement the unseen perspective.

• •

We introduce Physical Identity Inversion through model fine-tuning, effectively embedding personalized visual and physical identity into the model, thereby enabling accurate generation of identical human motions from the rear-view perspective.

• •

We present a patch-based denoising strategy for super-resolution video generation, effectively bridging the resolution gap between input and generated videos. This approach enables the subsequent reconstructed avatar model to render images with consistent granularity across varying viewpoints.

2 Related Work

2.1 Neural Rendering for Human Reconstruction

Neural rendering, as it effectively bridging the 3D assets with 2D images, has emerged as a powerful technique for reconstructing human figures directly from images. Many approaches served this problem as reconstructing the human avatar by leveraging shape priors [75, 65, 49, 36, 50] from parametric human body templates [40, 46] or learning the skinning fields according to the skeletons [61, 31, 62, 8]. The human dynamics is then decomposed into rigid motions driven by skeletons and non-rigid deformations predicted by a pose-conditioned neural network. On the other hand, this strategy unifies appearance information from different frames into a common 3D space, facilitating avatar creation and synthesis under novel views and novel poses. However, the quality of the reconstructed avatar is constrained by the ability to translate low-frequency pose parameters into high-frequency dynamic details. PoseVocab [32] decomposed the pose latent vector into per-joint embeddings for richer pose encoding. SLRF [90, 91] divide the entire radiance field into smaller local radiance fields, enabling better representativeness for local body part. Animatable Gaussians [33] and MeshAvatar [6] represented the pose parameters as the corresponding SMPL position map and employed the powerful 2D networks to encode it, achieving the SOTA in reconstructing the details.

Due to the high costs associated with studio-based multi-view data capture, several studies [82, 21, 61, 48] have explored reconstructing humans from monocular images or videos. Single-view scenarios are particularly challenging because inevitable errors in estimated poses can degrade 3D correspondences across frames. To address this issue, some research has focused on error correction networks [75, 25], while others have utilized trainable pose parameters [10, 24]. Recently, 3DGS is incorporated to further improve the training and inference efficiency [16, 17, 74, 14, 51, 60, 45, 27, 30, 53, 93, 42]. Nevertheless, in order to stabilize the rendering quality in novel view, which is invisible from input, these methods usually constrained the complexity of the pose-conditioned networks. This limitation may hinder the accurate reconstruction of dynamic details observed in the input images. In contrast, our approach regularize the avatar representation using the priors from generative models, facilitating both reconstruction and novel view synthesis.

2.2 Generative Prior for Human Avatars

With the development of generative models [57, 7, 76, 4, 68, 11], more and more methods [28, 22, 41, 2, 23, 13, 73, 29] seek to inject human priors from large models into human avatar reconstruction. They mainly leverage the power of generative models to iteratively optimize 3D representations. For example, some methods employ score distillation sampling (SDS) loss [52] to optimize 3D representations conditioned on text prompt [28, 22, 41, 69, 78, 18, 35, 20, 29], skeletons [19, 84, 64, 38], and densepose [86], normal maps. However, textual descriptions and 2D pose maps inherently lack precision in representing fine-grained geometric and texture details. Moreover, SDS primarily optimizes 3D parameters by enforcing distributional constraints, which can lead to issues such as oversaturation and excessive smoothing. Although some methods [3] alleviate the above problems by using variational score distillation (VSD) loss [72], the computational cost is higher.

Due to the inherent limitations of SDS loss, subsequent works [2, 23, 13, 1, 5] avoid using it and instead explicitly leverage generative models [57, 88, 63]. These approaches typically generate multi-view images or videos using a pretrained large model, followed by direct 3D reconstruction and optimization. Human-related priors are learned from 2D structures and conditions (e.g., rendered skeletons or surface normals). Similarly, HumanSplat [43] generates multi-view features within latent space and reconstructs the human body through feature aggregation. Recent approaches [34, 44, 77, 80, 87] extends these strategies to video-to-4D generation. However, the absence of explicit 3D structural modeling exacerbates inherent temporal and view inconsistencies in 2D generative models, often leading to flat or blurry results. L4GM [56] and GVFDiffusion [85] directly synthesize 4D content conditioned on the given video frames. Despite this, they primarily handle simple skinning-like motions and fail to capture complex dynamic details. Preserving the human identity while recovering fine-grained motions remains a significant challenge.

Our task formulation is similar to WonderHuman [73], as both leverage generative priors to reconstruct dynamic 3D human avatar from a monocular video. WonderHuman optimize 3D human with SDS loss in both canonical and observation spaces to ensure visual consistency. However, due to the oversaturation and excessive smoothing caused by SDS, the generated invisible regions lack fine details and exhibit noticeable inconsistencies.

2.3 Identity Adaptation

Maintaining identity consistency is crucial in digital humans, particularly when handling complex textures and dynamic movements. Most methods [15, 79, 66, 92] use a single reference image as input and design a reference network to embed the image into the backbone model. However, this approach demands a substantial amount of data for training and is computationally expensive. In addition, IP-Adapter [81] designs an effective and lightweight adapter to enable image prompt for pre-trained text-to-image diffusion models. Specifically, IP-Adapter modifies the cross-attention mechanism by separating cross-attention layers for text and image features. By freezing most parameters, image prompt can be learned by training only the image cross-attention layers. DreamBooth [58] can fine-tune the generative model using a few reference images, ensuring identity consistency. In details, DreamBooth fine-tunes a pretrained diffusion model to link a unique identifier to a specific subject, allowing it to be seamlessly embedded into different scenes. VideoComposer [67] directly concatenates the reference image with noise to achieve identity injection.

3 Method

3.1 Overview

Given a monocular sequence of human motions with the corresponding human poses, which could be estimated using existing tools [9, 70], our task is to reconstruct a high-fidelity 3DGS avatar of this subject. High-frequency dynamic details often necessitate a trade-off between accurate input-view reconstruction and generalizable novel view synthesis. To address this, we propose leveraging advanced video generative models to generate additional videos from alternate perspectives as pseudo-supervision. This approach could effectively regularize the 3D representation from overfitting. However, generating a video that replicates human motions from an alternative viewpoint presents significant challenges. It requires preserving the individual’s identity both visually and physically, ensuring that distinctive features and movement characteristics remain consistent across perspectives. Additionally, video generative models often face computational memory constraints that limit their output to low-resolution videos, typically below 1080p. It is crucial to bridge resolution gap between generated video and input video, to maintain visual fidelity and meet quality expectations. We propose physical identity inversion (Sec. 3.3) through model finetuning and super-resolution generation (Sec. 3.4) to tackle these challenges, respectively. An overview of our method is presented in Fig. 2.

3.2 Preliminaries

Expressive 3DGS Avatar Representation. 3D Gaussian Splatting [26] is a 3D point-based representation for efficient and realistic rendering. It’s represented by a set of 3D Gaussians, each of which is parameterized by its 3D center position $\boldsymbol{\mu}$, a covariance matrix $\boldsymbol{\Sigma}$, opacity $\alpha$ and color $\boldsymbol{c}$, and distributed as:

[TABLE]

where the covariance matrix is further parameterized by a rotation quaternion $\mathbf{q}$ and a 3D scaling vector $\mathbf{s}$ practically.

The rendering is conducted by splatting these 3D Gaussians onto the 2D plane, where the color of each pixel is computed by composition of the Gaussians overlapping on it. Following recent advancements on human avatars [33, 16], we represent the avatar as 2D Gaussian map, and utilize UNet $\mathcal{U}$ to produce the Gaussian maps under different poses. Specifically, the Gaussian points are anchored onto the SMPL template by orthogonally projecting the canonical mesh to the front plane and back plane. Each active pixel corresponds to a 3D Gaussian point. Given any human pose $\boldsymbol{\Theta}=\{\theta_{i}\}_{i=1}^{J}$, these points could be deformed using the transformation matrix from LBS, and other Gaussian attributes are acquired by feeding the front-back projected posed position map [33, 6] $\mathcal{P}_{f},\mathcal{P}_{b}$ to UNet:

[TABLE]

where $\mathcal{G}=(\Delta\boldsymbol{\mu},\mathbf{q},\mathbf{s},\alpha,\mathbf{c})$ consists of a pose-dependent point offset $\Delta\boldsymbol{\mu}$ to model non-rigid deformations.

Human4DiT. Human4DiT [59] is a DiT-based [47] approach for generating high-quality, 360-degree human videos with spatio-temporal coherence from a single image. By leveraging a hierarchical structure and the strength of DiT in capturing global features, it enables the synthesis of videos with strong 3D consistency. Specifically, Human4DiT proposes a hierarchical 4D transformer architecture that factorizes self-attention across views, time steps, and spatial dimensions. To enable precise control, dedicated modules are designed to embed camera parameters, temporal information, human motion, and identity. Additionally, Human4DiT collects large-scale multi-dimensional datasets, including 2D videos, multi-view videos, 3D data, and 4D data. By employing a multi-dimensional training strategy, the model effectively leverages these diverse data to enhance the generalization ability.

Human4DiT iteratively denoises a gaussian noise $\epsilon$ to obtain a clean latent representation $z$, which is then passed through a decoder to generate the final output. During training, the model is trained to predict the applied noise from the noisy latent $z_{t}$:

[TABLE]

where $\epsilon_{\theta}$ represents the denoising transformer, $t$ represents the timesteps, and $c_{\mathrm{ref}}$ and $c_{\Theta}$ represent the conditions of identity reference and human poses, respectively.

3.3 Physical Identity Inversion by Finetuning

In Human4DiT, the human identity is injected to the network through a CLIP [54] embedding $c_{\mathrm{ref}}$ extracted from a reference image. This approach has several limitations. On the one hand, the embedding conflates visual patterns from both the human subject and the background, leading to insufficient emphasis on the human identity. On the other hand, since the embedding is based on a single image, it provides an incomplete representation of the human identity, inadequately capturing the complex physical properties necessary for high-quality generation.

To overcome these limitations, we inject the human identity by fine-tuning the model using input video data. Specifically, we define a unique, learnable embedding, $c_{id}$, initialized from the CLIP embedding corresponding to the human subject, which enables a richer representation of identity beyond the constraints of a single-image reference. The fine-tuning is performed using the reconstruction loss analogous to Equation 3. Additionally, to preserve capability to generate multi-view consistent video, we alternate tuning the model between the input video and the multi-view dataset from Human4DiT. Notably, only the attention layers conditioned on identity are fine-tuned.

3.4 Back-view Generation with Super Resolution

In order to fully capture previously unseen details, we opted to generate the video from a rear perspective. In particular, based on the SMPL parameters of the first frame, we derived the new camera extrinsic matrix by rotating the original camera 180 degree around the axis passing through the root position and directed along the global orientation.

Human4DiT generates images at a standard resolution of $768\times 768$, which is considerably lower than the typical 2K resolution or higher found in most images. This discrepancy creates a pronounced resolution gap when transitioning from the front view to the back view. To address this, we instead generate images at twice the original resolution in both width and height. Enlarging the canvas won’t affect the decoding because of the shift invariance from VAE decoder. Consequently, we propose an algorithm that produces larger latent representations using a smaller, fixed-size diffusion denoising approach. Specifically, we employ a similar sliding window strategy to maintain spatial consistency in the generated images, as how other methods preserve temporal consistency in video generation. As illustrated in Fig. 4, at each denoising timestep the latent image is partitioned into nine overlapping patches; noise is predicted independently for each patch and then merged via a weighted sum. Correspondingly, during model fine-tuning (Sec. 3.3), we incorporate randomly cropped videos into the training dataset to enhance patch-based generation.

After generating the back-view video, we directly treat it as a real captured video for pseudo supervision.

4 Experiments

Dataset. We evaluated our method using the THuman4.0 dataset [90], a multi-view dataset with a resolution of $1330\times 1150$, featuring characters with rich textures and dynamic details. To assess our method on higher-resolution images, we collected an additional dataset, named Mono2K, comprising images at the resolution of $1500\times 2048$. For evaluation, we selected all three sequences (subject00, subject01 and subject02) from THuman4.0 and two typical sequences (Mono2K-male and Mono2K-female) from Mono2K. Each sequence includes manually selected video clips featuring turning motions to ensure full-body visibility. For fair comparison in novel view synthesis, we utilized ground truth SMPL-X poses fitted from multiple views as input. For methods requiring SMPL poses, we converted them manually using the official tool [46]. Foreground masks were obtained using Segment Anything 2 [55].

Baselines. We compared our method with several state-of-the-art approaches, including HumanNeRF [75], GaussianAvatar [16], and Animatable Gaussians [33]. Both GaussianAvatar and Animatable Gaussians are 3DGS-based methods for constructing human avatars, aiming to reconstruct dynamic details through 2D UNet architectures. GaussianAvatar is designed for monocular inputs, while Animatable Gaussians is tailored for multi-view inputs. Due to the training efficiency of NeRF-based methods, when evaluating HumanNeRF on the Mono2K dataset, we resized the images to $750\times 1024$.

The comparison is also conducted with state-of-the-art video-to-4D approaches, L4GM [56] and GVFDiffusion [85], both of which are designed for general objects. L4GM leverages the generated multi-view videos, whereas GVFDiffusion adopts a holistic encode–decode paradigm to directly generate 4D content. All methods are reproduced using their publicly available codebases.

Metrics. We conducted our evaluation using established image quality metrics: Peak Signal-to-Noise Ratio (PSNR), Structure Similarity Index Measure (SSIM) [71] and Learned Perceptual Image Patch Similarity (LPIPS) [89]. PSNR and SSIM are calculated over the entire image, with backgrounds set to white, while LPIPS is computed within the minimal square bounding box compassing the body.

4.1 Main Results

Fig. 3 showcases our reconstruction results by rendering the reconstructed subject from different angles of viewpoints. Our approach demonstrates the capability not only in recovering the fine-grained details from the input view, but also produce plausible rendering quality at novel views. These results prove the robustness and adaptability of our approach in complementing missing details and regularizing dynamic human representations.

4.2 Comparisons

We assess the effectiveness of our approach through evaluations on both input view reconstruction and novel view synthesis. The quantitative results are presented in Tab. 1. As illustrated in Fig. 5, by leveraging priors from a video generative model, our method significantly outperforms other state-of-the-art approaches. Due to its NeRF-based representation, HumanNeRF cannot accurately recover the fine details from the input, primarily producing dynamics through skinning. In contrast, both GaussianAvatar and Animatable Gaussians demonstrate the superior representational capabilities of 3DGS for precise reconstruction. However, they exhibit artifacts in unseen regions, likely due to the lack of effective regularization for 3DGS, even though GaussianAvatar was specifically designed for monocular input. Complementing the back view not only supplies additional reconstruction details but also regularizes the avatar representation, mitigating potential artifacts and enabling our method to achieve high-fidelity and high-quality reconstruction.

Moreover, as illustrated in the last example in Fig. 5, both HumanNeRF and GaussianAvatar are unable to accurately capture certain dynamic phenomena, such as the fluttering clothes, because their representations are tightly constrained to the human template. Our approach leverages video generation that rigorously preserves the subject’s physical identity while ensuring view consistency with the input, enabling the reconstruction of these intricate dynamic details. This capability distinguishes our method from prior approaches, which have struggled to achieve similar results.

We further compare our method with SOTA video-to-4D approaches. As shown in Fig. 6, both L4GM and GVFDiffusion struggle to reconstruct complex non-rigid dynamics present in the video. L4GM, due to the absence of human-body priors, fails to maintain consistent human identity across viewpoints and timestamps. GVFDiffusion, on the other hand, fails to encode the full range of motion details in the input video, leading to generated results that deviate substantially from the original content. In contrast, our approach benefits from both 3D human templates and generative priors, achieving high-fidelity dynamic reconstruction.

4.3 Ablation Study

**ID Finetuning. ** Fig. 7 illustrates the generation results of different finetuning strategies. (a) Without finetuning, Human4DiT generates identities with only approximate color similarity. (b) Fine-tuning only the identity embedding yields a generally accurate appearance but lacks precision in details, such as clothing wrinkles, due to insufficient incorporation of the subject’s dynamic characteristics. (c) Finetuning along with the Human4DiT model implicitly injects the physical properties to generate dynamics. This comprehensive fine-tuning leads to the most accurate reconstruction of the human identity, enhancing the quality of subsequent back-view video generation.

Super-resolution Generation. Fig. 8 presents the back-view generation results at different resolutions. As shown in (b) and (e), the absence of super-resolution leads to blurred clothing wrinkles, as diffusion models struggle to capture fine-grained local details. In contrast, our super-resolution approach not only enhances visual clarity and preserves intricate details but also generates high-resolution images that are crucial for improving the quality and robustness of avatar training.

5 Conclusion

We present a novel framework for high-fidelity dynamic human reconstruction from monocular video. By leveraging video generation to incorporate alternative viewpoints, our approach not only recovers high-frequency dynamic details from the input view but also supports novel view synthesis with equivalent quality. Furthermore, we enhance video generation by employing Physical Identity Inversion through model fine-tuning in conjunction with patch-based super-resolution techniques.

Limitations. Our approach is constrained by the current capabilities of the avatar model. As a pose-conditioned network, the avatar representation faces challenges in reconstructing data with pose-appearance one-to-many issue, a scenario commonly observed with loose clothing.

Appendix A Implementation Details

The avatar is finally optimized by:

[TABLE]

where $\mathcal{L}_{\mathrm{rgb}}$ is the L1 photometric loss between the rendered image and the ground truth image, $\mathcal{L}_{\mathrm{LPIPS}}$ is the perceptual loss [89], and $\mathcal{L}_{\mathrm{offset}},\mathcal{L}_{\mathrm{scale}}$ are regularization terms for the Gaussian points:

[TABLE]

[TABLE]

encouraging the point cloud close to the body surface and keeping the Gaussians from being spiky.

During training, the ID finetuning and Super-resolution back-view generation are conducted on a single NVIDIA A800 with 80GB GPU memory. The identity dynamics injection requires approximately 20k steps and 20 hours, followed by an additional 6 hours to generate the double-resolution video. After the data preparation, the avatar model is trained on a single NVIDIA RTX 4090 GPU. It takes around 50k steps and 4 hours to converge. The reconstructed avatar could be rendered at 8$\sim$10 FPS.

Appendix B Evaluation on the Generated Video

In this section, we present an evaluation of our generated videos. Table A reports the quantitative results on the THuman4.0 dataset, following the format of Table 1 in the main text. Since novel views are inherently unknown for our task, we additionally compute the Fréchet Inception Distance (FID) [12] to assess generative fidelity, which serves as a more suitable metric for novel view synthesis. Similar to LPIPS, the FID value is calculated within the minimal square bounding box enclosing the human region.

As shown in the top panel of the table, the resolution gap between the generative model and the 3D Gaussian model ($768$ v.s. $1150$) prevents the former from achieving competitive reconstruction quality with the latter. Supervising using the generated video leads to a slight degradation in input-view fidelity. However, for novel views, the generated video exhibits higher quality compared to previous avatar-based approaches. Notably, the evaluated novel views differ from the “back-view” employed in our method, and generating videos for these views incurs substantially higher computational cost than directly rendering our reconstructed 3DGS. These quantitative results confirm that our back-view supervision strategy effectively distills the generative priors into the avatar model, achieving significant improvements over previous state-of-the-art methods.

Appendix C More results

Due to the page limit of the main paper, we show more results in the supplementary materials. As shown in the Fig A, our method achieves high-fidelity human reconstruction with dynamic details and supports dynamically consistent image synthesis at novel views. To overcome the challenges for generating view-consistent and detailed results, we propose to leverage the advanced video generative model, Human4DiT, to generate the human motions from alternative perspective as an additional supervision signal. In addition, to solve the problem of insufficient resolution and inconsistent identity of Human4DiT, we inject the physical identity into the model through video fine-tuning and employ a patch-based denoising algorithm. Quantitative and qualitative experiments prove the effectiveness of the proposed method.

To better compare the experimental results, the quantitative comparisons for each subject are shown in Tab. B.

Appendix D Novel Pose Synthesis

After constructing the human avatar, we can directly deform the learned model to animate the character under arbitrary poses. Furthermore, a similar reconstruction strategy can be employed to enable novel pose synthesis. Specifically, given the novel poses, we generate front-view and back-view videos simultaneously and incorporate them as additional training data for the learned avatar. This approach leverages both the inherent human structural priors and the generative priors, similar to our reconstruction process.

Next, we present several synthesized results obtained using this strategy. A pose sequence from THuman4.0-subject02, which does not overlap with the training data, is selected as the test sequence. Figure B illustrates the synthesized results for all subjects in the THuman4.0 dataset, with each row corresponding to the poses at different timestamps. Our method is able to generate rich dynamic details that remain consistent with those observed in the training frames, which demonstrate that our Physical Identity Inversion strategy effectively captures the physical properties of the human body.

Appendix E In-the-wild Examples

In the main text, we evaluate all methods using self-rotating sequences to ensure fair comparisons. However, this condition is not necessary for our approach. Figure C presents reconstructed results on in-the-wild examples with only front-view frames observed. The example in the top row is taken from GaussianAvatar [16], while the remaining examples are sourced from YouTube videos that were also used in HumanNeRF [75]. These examples are included solely for academic purposes. Prior to processing the in-the-wild data in our pipeline, the estimated poses are refined using the same strategy as the first-stage training of GaussianAvatar. As shown in the figure, even without back-view observations, our method is capable of synthesizing a plausible full-body appearance.

Appendix F Potential Social Impacts

Our approach automates the creation of digital images representing any individual’s identity from monocular video, where the video can be effortlessly captured using a smartphone. However, this feature also carries the risk of producing fabricated motion sequences that the person never executed, a concern that must be thoroughly addressed before deployment.