StEP: Style-based Encoder Pre-training for Multi-modal Image Synthesis

TL;DR

This paper introduces StEP, a style-based encoder pre-training method for multi-modal image-to-image translation that improves generalization, fidelity, and training efficiency by modeling style variability independently of specific tasks.

Contribution

It proposes a novel style encoder pre-training approach that creates a versatile, expressive latent space for multi-modal I2I translation, simplifying training and enhancing results.

Findings

Achieves state-of-the-art results on six benchmarks.

Provides a more powerful style representation for image translation.

Simplifies training with a minimal loss setup.

Abstract

We propose a novel approach for multi-modal Image-to-image (I2I) translation. To tackle the one-to-many relationship between input and output domains, previous works use complex training objectives to learn a latent embedding, jointly with the generator, that models the variability of the output domain. In contrast, we directly model the style variability of images, independent of the image synthesis task. Specifically, we pre-train a generic style encoder using a novel proxy task to learn an embedding of images, from arbitrary domains, into a low-dimensional style latent space. The learned latent space introduces several advantages over previous traditional approaches to multi-modal I2I translation. First, it is not dependent on the target dataset, and generalizes well across multiple domains. Second, it learns a more powerful and expressive latent space, which improves the fidelity of style capture and transfer. The proposed style pre-training also simplifies the training objective and speeds up the training significantly. Furthermore, we provide a detailed study of the contribution of different loss terms to the task of multi-modal I2I translation, and propose a simple alternative to VAEs to enable sampling from unconstrained latent spaces. Finally, we achieve state-of-the-art results on six challenging benchmarks with a simple training objective that includes only a GAN loss and a reconstruction loss.