Sem-GAN: Semantically-Consistent Image-to-Image Translation

TL;DR

Sem-GAN introduces a semantically-aware image translation framework that maintains source image semantics while translating to the target domain, improving realism and segmentation performance in unpaired image-to-image translation tasks.

Contribution

It proposes a novel semantically-consistent GAN framework that enforces semantic preservation during unpaired image translation, addressing limitations of invertibility in multi-modal scenarios.

Findings

Sem-GAN improves translation quality by over 20% in FCN scores.

Semantic segmentation models trained on Sem-GAN translated images outperform other methods.

The framework effectively maintains semantic consistency in complex translation tasks.

Abstract

Unpaired image-to-image translation is the problem of mapping an image in the source domain to one in the target domain, without requiring corresponding image pairs. To ensure the translated images are realistically plausible, recent works, such as Cycle-GAN, demands this mapping to be invertible. While, this requirement demonstrates promising results when the domains are unimodal, its performance is unpredictable in a multi-modal scenario such as in an image segmentation task. This is because, invertibility does not necessarily enforce semantic correctness. To this end, we present a semantically-consistent GAN framework, dubbed Sem-GAN, in which the semantics are defined by the class identities of image segments in the source domain as produced by a semantic segmentation algorithm. Our proposed framework includes consistency constraints on the translation task that, together with the GAN loss and the cycle-constraints, enforces that the images when translated will inherit the appearances of the target domain, while (approximately) maintaining their identities from the source domain. We present experiments on several image-to-image translation tasks and demonstrate that Sem-GAN improves the quality of the translated images significantly, sometimes by more than 20% on the FCN score. Further, we show that semantic segmentation models, trained with synthetic images translated via Sem-GAN, leads to significantly better segmentation results than other variants.