Mask Based Unsupervised Content Transfer

TL;DR

This paper introduces a mask-based unsupervised content transfer method that disentangles shared and unique features between domains, enabling high-quality translation, content manipulation, and weakly-supervised segmentation.

Contribution

It proposes a novel mask generation approach for unsupervised content transfer that improves translation quality and enables content editing and segmentation with minimal supervision.

Findings

Achieves state-of-the-art translation quality and diversity.

Capable of content addition, removal, and editing across domains.

Enables weakly-supervised semantic segmentation with only class labels.

Abstract

We consider the problem of translating, in an unsupervised manner, between two domains where one contains some additional information compared to the other. The proposed method disentangles the common and separate parts of these domains and, through the generation of a mask, focuses the attention of the underlying network to the desired augmentation alone, without wastefully reconstructing the entire target. This enables state-of-the-art quality and variety of content translation, as demonstrated through extensive quantitative and qualitative evaluation. Our method is also capable of adding the separate content of different guide images and domains as well as remove existing separate content. Furthermore, our method enables weakly-supervised semantic segmentation of the separate part of each domain, where only class labels are provided. Our code is publicly available at https://github.com/rmokady/mbu-content-tansfer.