Human Multi-View Synthesis from a Single-View Model:Transferred Body and Face Representations

TL;DR

This paper introduces a novel multi-view human image synthesis framework that leverages transferred body and facial representations from a single-view pretrained model, overcoming dataset limitations and improving detail accuracy.

Contribution

It presents a new method combining transferred body and facial features with diffusion models for enhanced multi-view human synthesis from a single image.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Achieves more realistic and detailed multi-view human images.

Effectively extends 2D single-view knowledge to multi-view synthesis.

Abstract

Generating multi-view human images from a single view is a complex and significant challenge. Although recent advancements in multi-view object generation have shown impressive results with diffusion models, novel view synthesis for humans remains constrained by the limited availability of 3D human datasets. Consequently, many existing models struggle to produce realistic human body shapes or capture fine-grained facial details accurately. To address these issues, we propose an innovative framework that leverages transferred body and facial representations for multi-view human synthesis. Specifically, we use a single-view model pretrained on a large-scale human dataset to develop a multi-view body representation, aiming to extend the 2D knowledge of the single-view model to a multi-view diffusion model. Additionally, to enhance the model's detail restoration capability, we integrate transferred multimodal facial features into our trained human diffusion model. Experimental evaluations on benchmark datasets demonstrate that our approach outperforms the current state-of-the-art methods, achieving superior performance in multi-view human synthesis.