Human-Inspired Facial Sketch Synthesis with Dynamic Adaptation

TL;DR

This paper introduces HIDA, a human-inspired method for facial sketch synthesis that dynamically adapts to 3D geometry, appearance, and style, producing high-quality, multi-style sketches with precise control.

Contribution

The novel HIDA approach dynamically modulates neuron activations considering 3D geometry, appearance, and style, improving sketch quality and style control over previous methods.

Findings

HIDA outperforms previous methods on challenging faces.

It generates high-quality sketches in multiple styles.

It allows precise style control and generalizes well.

Abstract

Facial sketch synthesis (FSS) aims to generate a vivid sketch portrait from a given facial photo. Existing FSS methods merely rely on 2D representations of facial semantic or appearance. However, professional human artists usually use outlines or shadings to covey 3D geometry. Thus facial 3D geometry (e.g. depth map) is extremely important for FSS. Besides, different artists may use diverse drawing techniques and create multiple styles of sketches; but the style is globally consistent in a sketch. Inspired by such observations, in this paper, we propose a novel Human-Inspired Dynamic Adaptation (HIDA) method. Specially, we propose to dynamically modulate neuron activations based on a joint consideration of both facial 3D geometry and 2D appearance, as well as globally consistent style control. Besides, we use deformable convolutions at coarse-scales to align deep features, for generating abstract and distinct outlines. Experiments show that HIDA can generate high-quality sketches in multiple styles, and significantly outperforms previous methods, over a large range of challenging faces. Besides, HIDA allows precise style control of the synthesized sketch, and generalizes well to natural scenes and other artistic styles. Our code and results have been released online at: https://github.com/AiArt-HDU/HIDA.