DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN

TL;DR

DrawingInStyles introduces a spatially conditioned StyleGAN that enables precise, user-friendly portrait image generation and editing from sketches or semantic maps, improving control and quality over previous methods.

Contribution

We propose SC-StyleGAN, a novel spatially conditioned StyleGAN architecture, and a drawing interface that allows non-professionals to generate and edit high-quality face images easily.

Findings

Superior generation quality demonstrated through evaluations

Enhanced control with sketch and semantic map inputs

Positive user study confirming usability and expressiveness

Abstract

The research topic of sketch-to-portrait generation has witnessed a boost of progress with deep learning techniques. The recently proposed StyleGAN architectures achieve state-of-the-art generation ability but the original StyleGAN is not friendly for sketch-based creation due to its unconditional generation nature. To address this issue, we propose a direct conditioning strategy to better preserve the spatial information under the StyleGAN framework. Specifically, we introduce Spatially Conditioned StyleGAN (SC-StyleGAN for short), which explicitly injects spatial constraints to the original StyleGAN generation process. We explore two input modalities, sketches and semantic maps, which together allow users to express desired generation results more precisely and easily. Based on SC-StyleGAN, we present DrawingInStyles, a novel drawing interface for non-professional users to easily produce high-quality, photo-realistic face images with precise control, either from scratch or editing existing ones. Qualitative and quantitative evaluations show the superior generation ability of our method to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.