Maximum Spatial Perturbation Consistency for Unpaired Image-to-Image Translation

TL;DR

This paper introduces a universal regularization called maximum spatial perturbation consistency (MSPC) for unpaired image-to-image translation, addressing limitations of existing constraints by better handling spatial variations and reducing content distortion.

Contribution

The paper proposes MSPC, a novel regularization enforcing commutativity between spatial perturbation and translation, with adversarial training components, improving I2I translation quality and robustness.

Findings

Outperforms state-of-the-art methods on most benchmarks.

Effective in handling spatial variations like object size and distortion.

Validated on a new front face to profile face dataset.

Abstract

Unpaired image-to-image translation (I2I) is an ill-posed problem, as an infinite number of translation functions can map the source domain distribution to the target distribution. Therefore, much effort has been put into designing suitable constraints, e.g., cycle consistency (CycleGAN), geometry consistency (GCGAN), and contrastive learning-based constraints (CUTGAN), that help better pose the problem. However, these well-known constraints have limitations: (1) they are either too restrictive or too weak for specific I2I tasks; (2) these methods result in content distortion when there is a significant spatial variation between the source and target domains. This paper proposes a universal regularization technique called maximum spatial perturbation consistency (MSPC), which enforces a spatial perturbation function (T ) and the translation operator (G) to be commutative (i.e., TG = GT ). In addition, we introduce two adversarial training components for learning the spatial perturbation function. The first one lets T compete with G to achieve maximum perturbation. The second one lets G and T compete with discriminators to align the spatial variations caused by the change of object size, object distortion, background interruptions, etc. Our method outperforms the state-of-the-art methods on most I2I benchmarks. We also introduce a new benchmark, namely the front face to profile face dataset, to emphasize the underlying challenges of I2I for real-world applications. We finally perform ablation experiments to study the sensitivity of our method to the severity of spatial perturbation and its effectiveness for distribution alignment.