TurboPortrait3D: Single-step diffusion-based fast portrait novel-view synthesis

TL;DR

TurboPortrait3D is a fast, diffusion-based method for high-quality, multi-view consistent portrait synthesis from a single image, improving visual fidelity and identity preservation over existing approaches.

Contribution

The paper introduces a novel single-step diffusion refinement process for 3D portrait synthesis that enhances quality while maintaining low latency.

Findings

Outperforms state-of-the-art in portrait view synthesis

Produces multi-view consistent and high-quality images

Operates efficiently with low computational cost

Abstract

We introduce TurboPortrait3D: a method for low-latency novel-view synthesis of human portraits. Our approach builds on the observation that existing image-to-3D models for portrait generation, while capable of producing renderable 3D representations, are prone to visual artifacts, often lack of detail, and tend to fail at fully preserving the identity of the subject. On the other hand, image diffusion models excel at generating high-quality images, but besides being computationally expensive, are not grounded in 3D and thus are not directly capable of producing multi-view consistent outputs. In this work, we demonstrate that image-space diffusion models can be used to significantly enhance the quality of existing image-to-avatar methods, while maintaining 3D-awareness and running with low-latency. Our method takes a single frontal image of a subject as input, and applies a feedforward image-to-avatar generation pipeline to obtain an initial 3D representation and corresponding noisy renders. These noisy renders are then fed to a single-step diffusion model which is conditioned on input image(s), and is specifically trained to refine the renders in a multi-view consistent way. Moreover, we introduce a novel effective training strategy that includes pre-training on a large corpus of synthetic multi-view data, followed by fine-tuning on high-quality real images. We demonstrate that our approach both qualitatively and quantitatively outperforms current state-of-the-art for portrait novel-view synthesis, while being efficient in time.