End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning

TL;DR

This paper introduces a fully convolutional neural network for automatic photo-to-sketch transformation, improving face sketch generation and verification by preserving details and identity features.

Contribution

It presents a novel end-to-end convolutional model that directly maps face photos to sketches, incorporating discriminative regularization for better identity preservation.

Findings

Outperforms state-of-the-art methods in sketch generation

Enhances face verification accuracy with generated sketches

Efficient inference due to fully convolutional architecture

Abstract

Sketch-based face recognition is an interesting task in vision and multimedia research, yet it is quite challenging due to the great difference between face photos and sketches. In this paper, we propose a novel approach for photo-sketch generation, aiming to automatically transform face photos into detail-preserving personal sketches. Unlike the traditional models synthesizing sketches based on a dictionary of exemplars, we develop a fully convolutional network to learn the end-to-end photo-sketch mapping. Our approach takes whole face photos as inputs and directly generates the corresponding sketch images with efficient inference and learning, in which the architecture are stacked by only convolutional kernels of very small sizes. To well capture the person identity during the photo-sketch transformation, we define our optimization objective in the form of joint generative-discriminative minimization. In particular, a discriminative regularization term is incorporated into the photo-sketch generation, enhancing the discriminability of the generated person sketches against other individuals. Extensive experiments on several standard benchmarks suggest that our approach outperforms other state-of-the-art methods in both photo-sketch generation and face sketch verification.