Multimodal Unsupervised Image-to-Image Translation

TL;DR

This paper introduces MUNIT, a framework for diverse, multimodal unsupervised image-to-image translation that decomposes images into content and style codes, enabling controllable and varied translations without paired data.

Contribution

The paper proposes a novel multimodal framework that separates content and style for unsupervised image translation, allowing diverse and controllable outputs, unlike previous deterministic methods.

Findings

MUNIT outperforms state-of-the-art methods in diverse image translation tasks.

The framework enables style control through example images.

Theoretical analysis supports the effectiveness of content-style disentanglement.

Abstract

Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to the state-of-the-art approaches further demonstrates the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at https://github.com/nvlabs/MUNIT