Unsupervised Contrastive Photo-to-Caricature Translation based on Auto-distortion

TL;DR

This paper introduces an unsupervised contrastive learning framework for photo-to-caricature translation that combines style rendering and geometric deformation, producing artistic caricatures without paired data.

Contribution

It proposes a novel contrastive style loss and a distortion prediction module for unpaired, bidirectional photo-to-caricature translation, enhancing style and exaggeration effects.

Findings

Effective in generating hand-drawn style caricatures

Outperforms existing methods in quality and exaggeration

Works with unpaired photo and caricature datasets

Abstract

Photo-to-caricature translation aims to synthesize the caricature as a rendered image exaggerating the features through sketching, pencil strokes, or other artistic drawings. Style rendering and geometry deformation are the most important aspects in photo-to-caricature translation task. To take both into consideration, we propose an unsupervised contrastive photo-to-caricature translation architecture. Considering the intuitive artifacts in the existing methods, we propose a contrastive style loss for style rendering to enforce the similarity between the style of rendered photo and the caricature, and simultaneously enhance its discrepancy to the photos. To obtain an exaggerating deformation in an unpaired/unsupervised fashion, we propose a Distortion Prediction Module (DPM) to predict a set of displacements vectors for each input image while fixing some controlling points, followed by the thin plate spline interpolation for warping. The model is trained on unpaired photo and caricature while can offer bidirectional synthesizing via inputting either a photo or a caricature. Extensive experiments demonstrate that the proposed model is effective to generate hand-drawn like caricatures compared with existing competitors.