High Resolution Zero-Shot Domain Adaptation of Synthetically Rendered Face Images

TL;DR

This paper introduces a novel zero-shot method to convert synthetically generated face images into high-resolution photorealistic images using a pretrained StyleGAN2, eliminating the need for synthetic training data.

Contribution

It presents the first algorithm capable of high-resolution (1K) face image translation from synthetic to photorealistic without synthetic training data.

Findings

Achieves 1K resolution photorealistic face images from synthetic inputs.

No synthetic training data required for the domain adaptation.

Significant improvement in visual realism over previous methods.

Abstract

Generating photorealistic images of human faces at scale remains a prohibitively difficult task using computer graphics approaches. This is because these require the simulation of light to be photorealistic, which in turn requires physically accurate modelling of geometry, materials, and light sources, for both the head and the surrounding scene. Non-photorealistic renders however are increasingly easy to produce. In contrast to computer graphics approaches, generative models learned from more readily available 2D image data have been shown to produce samples of human faces that are hard to distinguish from real data. The process of learning usually corresponds to a loss of control over the shape and appearance of the generated images. For instance, even simple disentangling tasks such as modifying the hair independently of the face, which is trivial to accomplish in a computer graphics approach, remains an open research question. In this work, we propose an algorithm that matches a non-photorealistic, synthetically generated image to a latent vector of a pretrained StyleGAN2 model which, in turn, maps the vector to a photorealistic image of a person of the same pose, expression, hair, and lighting. In contrast to most previous work, we require no synthetic training data. To the best of our knowledge, this is the first algorithm of its kind to work at a resolution of 1K and represents a significant leap forward in visual realism.