Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation

TL;DR

This paper introduces SSSP, a novel method for converting simple sketches into 3D-aware, stereoscopic portraits using a tri-plane generative model, enhancing realism and user accessibility.

Contribution

The paper proposes a new 3D-aware sketch-to-portrait framework with sketch-aware constraints and a vector quantized module for layman-friendly input, advancing 3D portrait generation from sketches.

Findings

Produces high-quality, sketch-matched 3D portraits

Enhances detail correspondence with novel constraints

User study shows system is highly preferred

Abstract

Creating the photo-realistic version of people sketched portraits is useful to various entertainment purposes. Existing studies only generate portraits in the 2D plane with fixed views, making the results less vivid. In this paper, we present Stereoscopic Simplified Sketch-to-Portrait (SSSP), which explores the possibility of creating Stereoscopic 3D-aware portraits from simple contour sketches by involving 3D generative models. Our key insight is to design sketch-aware constraints that can fully exploit the prior knowledge of a tri-plane-based 3D-aware generative model. Specifically, our designed region-aware volume rendering strategy and global consistency constraint further enhance detail correspondences during sketch encoding. Moreover, in order to facilitate the usage of layman users, we propose a Contour-to-Sketch module with vector quantized representations, so that easily drawn contours can directly guide the generation of 3D portraits. Extensive comparisons show that our method generates high-quality results that match the sketch. Our usability study verifies that our system is greatly preferred by user.