SemST: Semantically Consistent Multi-Scale Image Translation via Structure-Texture Alignment

TL;DR

SemST is a novel unsupervised image translation method that maintains semantic consistency across domains by aligning structure and texture at multiple scales, improving high-resolution domain adaptation.

Contribution

Introduces SemST, a multi-scale contrastive learning approach that reduces semantic distortion and enhances image translation quality, especially for high-resolution images.

Findings

SemST effectively reduces semantic distortion in image translation.

Achieves state-of-the-art performance in unsupervised I2I translation.

Preliminary results show SemST benefits semantic segmentation pre-training.

Abstract

Unsupervised image-to-image (I2I) translation learns cross-domain image mapping that transfers input from the source domain to output in the target domain while preserving its semantics. One challenge is that different semantic statistics in source and target domains result in content discrepancy known as semantic distortion. To address this problem, a novel I2I method that maintains semantic consistency in translation is proposed and named SemST in this work. SemST reduces semantic distortion by employing contrastive learning and aligning the structural and textural properties of input and output by maximizing their mutual information. Furthermore, a multi-scale approach is introduced to enhance translation performance, thereby enabling the applicability of SemST to domain adaptation in high-resolution images. Experiments show that SemST effectively mitigates semantic distortion and achieves state-of-the-art performance. Also, the application of SemST to domain adaptation (DA) is explored. It is demonstrated by preliminary experiments that SemST can be utilized as a beneficial pre-training for the semantic segmentation task.