Leveraging in-domain supervision for unsupervised image-to-image translation tasks via multi-stream generators

TL;DR

This paper introduces a novel multi-stream generator architecture and segmentation-based regularization to improve unsupervised image-to-image translation by leveraging in-domain semantic priors, especially in complex urban scenes.

Contribution

It proposes a new architecture and loss function that incorporate semantic segmentation to enhance translation quality in unsupervised I2I tasks.

Findings

Superior quality in day-to-night urban image translation

Improved robustness through segmentation-based regularization

Enhanced downstream detection performance with augmented data

Abstract

Supervision for image-to-image translation (I2I) tasks is hard to come by, but bears significant effect on the resulting quality. In this paper, we observe that for many Unsupervised I2I (UI2I) scenarios, one domain is more familiar than the other, and offers in-domain prior knowledge, such as semantic segmentation. We argue that for complex scenes, figuring out the semantic structure of the domain is hard, especially with no supervision, but is an important part of a successful I2I operation. We hence introduce two techniques to incorporate this invaluable in-domain prior knowledge for the benefit of translation quality: through a novel Multi-Stream generator architecture, and through a semantic segmentation-based regularization loss term. In essence, we propose splitting the input data according to semantic masks, explicitly guiding the network to different behavior for the different regions of the image. In addition, we propose training a semantic segmentation network along with the translation task, and to leverage this output as a loss term that improves robustness. We validate our approach on urban data, demonstrating superior quality in the challenging UI2I tasks of converting day images to night ones. In addition, we also demonstrate how reinforcing the target dataset with our augmented images improves the training of downstream tasks such as the classical detection one.