Memory-guided Unsupervised Image-to-image Translation

TL;DR

This paper introduces a class-aware memory network for unsupervised image-to-image translation that effectively captures local style variations at the instance level, outperforming existing methods in handling multiple objects.

Contribution

The authors propose a novel memory-augmented framework that explicitly models class-wise style variations without requiring object detectors during inference.

Findings

Outperforms recent instance-level translation methods.

Achieves state-of-the-art results on benchmark datasets.

Effectively models local style variations across classes.

Abstract

We present a novel unsupervised framework for instance-level image-to-image translation. Although recent advances have been made by incorporating additional object annotations, existing methods often fail to handle images with multiple disparate objects. The main cause is that, during inference, they apply a global style to the whole image and do not consider the large style discrepancy between instance and background, or within instances. To address this problem, we propose a class-aware memory network that explicitly reasons about local style variations. A key-values memory structure, with a set of read/update operations, is introduced to record class-wise style variations and access them without requiring an object detector at the test time. The key stores a domain-agnostic content representation for allocating memory items, while the values encode domain-specific style representations. We also present a feature contrastive loss to boost the discriminative power of memory items. We show that by incorporating our memory, we can transfer class-aware and accurate style representations across domains. Experimental results demonstrate that our model outperforms recent instance-level methods and achieves state-of-the-art performance.