Image-to-image Transformation with Auxiliary Condition

TL;DR

This paper introduces Label-CycleGAN, an enhanced image-to-image transformation method that incorporates label information to better handle pose and shape variations, improving transformation quality especially with imbalanced data.

Contribution

The paper proposes Label-CycleGAN, a novel extension of CycleGAN that integrates label information for more accurate and pose-aware image transformations.

Findings

Improved transformation quality on SVHN to MNIST digit images.

Effective handling of pose and shape variations in simulated to real image transformation.

Demonstrated robustness with imbalanced training data.

Abstract

The performance of image recognition like human pose detection, trained with simulated images would usually get worse due to the divergence between real and simulated data. To make the distribution of a simulated image close to that of real one, there are several works applying GAN-based image-to-image transformation methods, e.g., SimGAN and CycleGAN. However, these methods would not be sensitive enough to the various change in pose and shape of subjects, especially when the training data are imbalanced, e.g., some particular poses and shapes are minor in the training data. To overcome this problem, we propose to introduce the label information of subjects, e.g., pose and type of objects in the training of CycleGAN, and lead it to obtain label-wise transforamtion models. We evaluate our proposed method called Label-CycleGAN, through experiments on the digit image transformation from SVHN to MNIST and the surveillance camera image transformation from simulated to real images.