ColorMapGAN: Unsupervised Domain Adaptation for Semantic Segmentation Using Color Mapping Generative Adversarial Networks

TL;DR

This paper introduces ColorMapGAN, a simple yet effective unsupervised domain adaptation method for semantic segmentation of satellite images, which transforms spectral distributions to improve accuracy across different geographic locations.

Contribution

The paper proposes ColorMapGAN, a novel GAN architecture that uses simple color transformations without convolutional layers for domain adaptation in spectral image segmentation.

Findings

Outperforms existing methods in accuracy.

Reduces computational complexity.

Effectively adapts spectral distributions across domains.

Abstract

Due to the various reasons such as atmospheric effects and differences in acquisition, it is often the case that there exists a large difference between spectral bands of satellite images collected from different geographic locations. The large shift between spectral distributions of training and test data causes the current state of the art supervised learning approaches to output unsatisfactory maps. We present a novel semantic segmentation framework that is robust to such shift. The key component of the proposed framework is Color Mapping Generative Adversarial Networks (ColorMapGAN), which can generate fake training images that are semantically exactly the same as training images, but whose spectral distribution is similar to the distribution of the test images. We then use the fake images and the ground-truth for the training images to fine-tune the already trained classifier. Contrary to the existing Generative Adversarial Networks (GANs), the generator in ColorMapGAN does not have any convolutional or pooling layers. It learns to transform the colors of the training data to the colors of the test data by performing only one element-wise matrix multiplication and one matrix addition operations. Thanks to the architecturally simple but powerful design of ColorMapGAN, the proposed framework outperforms the existing approaches with a large margin in terms of both accuracy and computational complexity.

Tables

Table 1. TABLE I: The statistics for the luxcarta data set.

City	$\#$ of patches	Area (km2)	Class frequency (%)
			building	road	tree
Bad Ischl	457	27.71	5.51	6.03	35.38
Villach	749	43.59	9.26	10.63	19.91
Béziers	407	25.75	19.09	17.62	10.91
Roanne	384	25.84	18.44	8.33	14.78

Table 2. TABLE II: IoU scores for the test cities in pair 1.

Method			Training: Bad Ischl, Test: Villach				Training: Villach, Test: Bad Ischl
			building	road	tree	Overall	building	road	tree	Overall
U-net			23.61	0.91	40.53	21.68	5.84	0.24	0.50	2.19
AdaptSegNet Single[12]			6.01	4.37	10.43	6.94	3.06	2.71	10.23	5.33
AdaptSegNet Multi[12]			24.59	9.02	56.08	29.86	14.26	4.46	24.66	14.46
Our proposed	framework with	CycleGAN[26]	43.03	28.96	68.86	46.95	43.62	38.69	71.68	51.33
		UNIT[27]	30.86	15.84	63.00	36.57	19.29	36.83	35.57	30.56
		MUNIT[28]	0.02	1.38	47.23	16.21	6.20	0.13	0.05	2.13
		DRIT[29]	0.01	3.96	8.72	4.23	0.00	10.19	0.01	3.40
		Gray world[61]	25.19	26.43	56.15	35.92	29.55	24.80	46.41	33.58
		Histogram matching[47]	24.95	29.34	61.59	38.63	6.45	0.92	1.28	2.88
		ColorMapGAN (ours)	48.47	37.82	58.92	48.40	49.16	41.75	59.84	50.25

Table 3. TABLE III: IoU scores for the test cities in pair 2.

Method			Training: Béziers, Test: Roanne				Training: Roanne, Test: Béziers
			building	road	tree	Overall	building	road	tree	Overall
U-net			26.13	11.16	7.79	15.03	19.85	0.00	0.00	6.62
AdaptSegNet Single[12]			6.61	11.05	3.37	7.01	11.07	4.19	3.71	6.32
AdaptSegNet Multi[12]			22.42	5.87	17.84	15.37	24.27	10.88	10.45	15.20
Our proposed	framework with	CycleGAN[26]	18.19	24.28	0.19	14.22	9.92	16.00	0.54	8.82
		UNIT[27]	41.99	38.47	0.39	26.95	29.99	39.19	3.11	24.10
		MUNIT[28]	10.17	1.66	0.31	4.05	7.49	0.84	0.13	2.82
		DRIT[29]	42.16	41.77	1.18	28.37	25.73	36.54	0.68	20.98
		Gray world[61]	51.47	40.42	18.25	36.71	14.61	31.32	21.99	22.64
		Histogram matching[47]	18.64	9.87	4.81	11.11	20.63	0.02	0.00	6.88
		ColorMapGAN (ours)	55.60	44.66	28.39	42.88	47.12	35.18	21.91	34.74

Table 4. TABLE IV: Training times for generating fake cities.

Method	Training time for 1 Iter. (secs.)
CycleGAN [26]	0.11
UNIT [27]	0.47
MUNIT [28]	0.45
DRIT [29]	0.29
ColorMapGAN	0.05

Table 5. TABLE V: Execution times for generating fake cities.

City	Execution time (seconds)
	Gray world[61]	Histogram matching[47]
Bad Ischl	1.46	19.77
Villach	1.89	26.78
Béziers	1.37	18.05
Roanne	1.24	20.32

Equations

$$\underset{\begin{array}{c}D\end{array}}{\max }V(D)={\mathbb{E}}_{x\sim p(x)}\text{log}[D(x)]+{\mathbb{E}}_{z\sim p(z)}[\text{log}(1-D(G(z)))],$$

$$\underset{\begin{array}{c}G\end{array}}{\min }V(G)={\mathbb{E}}_{z\sim p(z)}[\text{log}(1-D(G(z)))].$$

$$\begin{array}{r}\underset{G}{\min }\underset{D}{\max }V(D,G)={\mathbb{E}}_{x\sim p(x)}\text{log}D(x)+\\ {\mathbb{E}}_{z\sim p_{(z)}}\text{log}(1-D(G(z))).\end{array}$$

$$RGB=R\text{bigtimes}G\text{bigtimes}B,$$

$$R^{\prime }{G}^{\prime }{B}^{\prime }=RGB\circ W+K,$$

$$I=r\times 256\times 256+g\times 256+b,$$

$$R^{\prime }{G}^{\prime }{B}^{\prime }[I]=RGB[I]\circ W[I]+K[I],$$

$$DN(p)=\lfloor (p+1)\times 127.5\rfloor ,$$

$$\underset{\begin{array}{c}D\end{array}}{\min }V(D)={\mathbb{E}}_{x\sim p(x)}[(D(x)-1{)}^2]+{\mathbb{E}}_{y\sim p(y)}[(D(G(y)){)}^2].$$

$$\underset{\begin{array}{c}G\end{array}}{\min }V(G)={\mathbb{E}}_{y\sim p(y)}[(D(y)-1{)}^2].$$

Full text

I Introduction

With the continuous proliferation and improvement of satellite sensors, numerous new generation satellite missions have been created, which has made it possible to collect huge amounts of data. The massive satellite data have introduced new challenges to the remote sensing community. Semantic segmentation or dense labeling is the task of assigning a thematic label to each pixel in the image. Without a doubt, among the challenges that the remote sensing community is facing today, dense labeling of the satellite images is one of the most important one, as a good solution for this problem is of paramount importance to generate and automatically update the maps.

Over the last few years, convolutional neural networks (CNNs) have become the most commonly used tool for the task of semantic segmentation. In particular U-net[1] and its variants [2, 3, 4] are receiving a growing attention due to their great success in different domains such as medical imaging and remote sensing. The main limitation of the aforementioned CNNs is their extreme sensitivity to the training data. Although they perform very well when both training and test data come from the same distribution [5], their performance severely decreases when there is a large difference between spectral bands of the training and the test images. Considering that nowadays new generation satellites with a short revisit time acquire huge amounts of images from different parts of the world, one cannot assume that the distributions of the images are always similar. In addition, depending on when and where the data are collected, a large intra-class variability might be encountered in remote sensing images. For instance, color distributions of the same objects may greatly differ in the images captured in different times of the day as a result of the illumination difference. Similarly, because of the atmospheric effects, in some cases, even the images collected by the same satellite sensors might have very different radiometry, which makes the segmentation task even harder. Fig. 1 illustrates such an example, where color distributions of the training and the test data collected by the same satellite are noticeably different. As shown in the figure, even a well trained U-net, which is considered as the state of the art in semantic segmentation of satellite images[6], generates a poor map.

The unsupervised domain adaptation assumes that annotations for any part of the test data are not available, and aims at generating high quality segmentations even when there is a large domain shift between the training and the test images. In such a setting, in order to increase the generalization capabilities of the CNNs, one of the simplest and most common methods is to diversify the training data by applying various data augmentation techniques such as gamma correction and random contrast change [7, 8]. However, even though these techniques help the CNNs to generalize slightly better, when spectral difference between the training and the test data is huge, the improvement is usually insufficient. Although a good solution for domain adaptation is indispensable for various applications, this problem has not been investigated intensively in the field of remote sensing. The limitations pointed out in this section have motivated us to design a methodology for this problem.

I-A Related Work

In this section, we summarize the existing approaches on domain adaptation in the fields of computer vision and remote sensing.

Computer Vision: The majority of the proposed approaches in the literature are based on the idea of aligning distributions of both training and test images in a common space using generative adversatial networks (GAN) so that the trained classifier could segment the test images well. FCNs in the Wild [9] reduces the global distribution difference between the training and the test images by minimizing an adversarial loss. Aligning the distributions of the extracted features from both the training and the test images is also a common approach [10, 11]. The distribution alignment could also be performed in multiple layers of the network instead of only the final output [12, 13]. Hong et al. add random noise to the training images [14]. They observe that perturbing the training images helps the GAN to adapt better to the test data. Saito et al. introduces a maximum classifier discrepancy based approach, which alings the distributions by making use of class-specific decision boundaries [15]. The framework presented by Chen et al. utilizes Google Street View to collect unannotated images and to use their features. Curriculum Domain Adaptation (CDA) [16] performs the alignment both globally and on the generated super-pixels. Reality Oriented Adaptation (ROAD) [17] comprises two loss functions. The first one forces the weights of the trained model from the training images with the weights of the pre-trained model from ImageNet [18] to be as similar as possible. The second one handles the spatial aware domain adaptation by dividing the training and the test images into grids and minimizing distance between the grids of the training and the test images.

There have been attempts to solve the domain adaptation problem by regularizing or normalizing specific layers of the network, or by self-learning. The Fully Convolutional Tri-branch network (FCTN)[19] is one of the methods that falls into the category of self-learning. The presented network architecture consists of one encoder and three decoders. Two of the decoders pseudo-label the test image, the other one learns from the pseudo-labels and the test image. A class-balanced self learning approach is proposed in CBST[20]. Introducing new normalization method, regularization technique, or new loss functions that are specific for domain adaptation problem is invesigated in [21, 22, 23, 24]. Romijnders et al. discuss the limitations of the traditional normalization methods such as batch normalization, and propose a new domain agnostic normalization layer that is more suitable for domain adaptation[21]. Saito et al. introduce a new adversarial dropout regularization technique[22]. The IBN-Net [23] combines the batch normalization with the instance normalization [25] to increase the generalization capability. Zhu et al. proposes a new conservative loss [24].

Another way to approach the unsupervised domain adaptation problem is to perform image to image translation (I2I) between a source and a target domain. The I2I approaches aim at generating fake source images that are statistically indistinguishable from the target images. The existing I2I approaches in the literature can perform one to one [26, 27] or many to many [28, 29] translations. If one can generate fake training images, which are style-wise consistent with the test images and semantically consistent with the original training images, the fake images could be used to train a model from scratch or to fine-tune the already trained model on the original data. The main drawback here is that usually the fake training images generated by the I2I approaches contain artificial objects and artifacts, which do not exist in the original images. Hence, annotations for the original training images and the generated fake training images do not match. As a result, the model learns wrong information. To overcome this limitation, CyCADA [30] segments the original and the fake training images using the classifier trained on the original training data, and minimizes cross entropy loss between the segmentations. However, if the domain shift between the training and the test data is large, the classifier trained from the original training images will segment the fake training image poorly. If the segmentation for the training images is very good, but the predicted map for the fake training images is extremely noisy, we cannot expect adding such a loss to prevent artificial objects appearing. Another way to enrich the training data could be to perform neural style transfer, where content of an image and style of another image are combined [31, 32, 33, 34]. However, they also cannot guarantee that the semantic structures will be preserved.

Remote sensing: As thoroughly explained in the overview paper by Tuia et al. [35], the domain adaptation methods in the field of remote sensing can be divided into four categories: selection of invariant features, adaptation of classifiers, active learning, and adaptation of the data distributions. The main goal of the methods falling into the first category is to find a subset of features from the training data, which is representative for the test data. For instance, Bruzzone et al. present a methodology [36] aiming at selecting the features that are both invariant to the test data and discriminative for the classes of interest by defining a new criterion function quantifying both tasks. Persello et al. describe a method that solves the same selection problem by a kernel based method [37].

In order to adapt the classifier to the test data, the existing approaches usually either choose to adapt the classifier to the unlabeled test data directly, or favor an active learning approach, where the annotator labels a small number of representative samples from the test data to update the already trained classifier. Bruzzone et al. describe an approach that updates the parameters of the previously trained classifier on the unlabeled test data via the expectation-maximization (EM) algorithm [38]. This approach has then been extended to another framework, where a cascade classification operation is performed [39]. In [40], multiple cascade classifiers are used. Bruzzone and Marconcini propose the domain adaptation support vector machine (DASVM) [41]. If adapting the classifier by an active learning approach is preferred, we can obviously expect to get better results, since some portion of the test data is provided to modify the classifier. On the other hand, the selection of the samples and labeling them manually can be too costly. In the approach proposed by Persello and Bruzzone [42], the classifier iteratively learns from a small number of newly added samples from the test data and removes some of the training samples whose distribution does not fit with the distribution of the test data. The SVM based method presented by Matasci et al. also follows a similar approach [43]. Another kernel based active learning approach is introduced by Deng et al. [44]. The usage of neural networks for active learning is studied in [45].

The adaptation of the data distributions is a common method. To do this, one can either align the distributions of the training and the test data in a common space, or align the distribution of the training data to the distribution of the test data. In both cases, we expect the model trained from the aligned training data to generate better segmentation. The aforementioned alignment can be performed by histogram matching [46], [47], graph matching [48], kernel principal component analysis (KPCA) [49], color constancy algorithms [50], or minimizing the statistical distance between the training and the test data [51] by the maximum mean discrepancy (MMD) [52]. Ma et al. correct the domain shift between the training and the test data for each class by centroid and covariance alignment [53]. Gross et al. propose a nonlinear feature normalization method for the alignment [54]. A new methodology performing tensor alignment is presented in [55]. Recently, Courty et al. have used the optimal transport for domain adaptation [56, 57]. The optimal transport for domain adaptation of remote sensing data is studied in [58]. Besides, after GANs had become popular in the field of computer vision, some papers have studied their application to remote sensing. For instance, Benjdira et al. [59] use CycleGAN [26] to generate fake training data that resemble test data. The generated fake training data are then used to fine-tune the already trained model from the original training data. However, unlike the computer vision benchmarks, remote sensing images contain a lot of heterogeneous and complex structures. As a result, the quality of the fake remote sensing images generated by CycleGAN is usually not as good as desired.

I-B Contributions

We study the problem of unsupervised domain adaptation for semantic segmentation, where training and test images are collected from completely different geographic locations, they have significant spectral distribution difference, and annotations for any part of the test images are not available. The way we approach the problem methodologically resembles to the image to image translation (I2I) approaches; we generate fake training images as if they came from the distribution of the test images. Instead of working on images acquired over separate geographic extents, if we had paired training and test images (e.g., images acquired from exactly the same geographic extent but in different day time), the problem would be easier, we could perform paired I2I [60].

The main challenge here is to generate fake training images that look like test images without impairing the semantic meaning in the original training images, even when the training and the test images are unpaired. If the semantic structures of the original training images are preserved in the fake training images, we can utilize the annotations for the original training images as well as the fake training images to fine-tune the already trained model from the original training data. When generating fake training images with the style of the test images, most of the I2I approaches presented in Sec. I-A fail to preserve the exact structures of the original training images, especially when we deal with satellite images containing a lot of complex and heterogeneous objects. Thus, the fake training images and the annotations do not mach, which leads the model to learn incorrect information.

Contributions of this work are as follows:

1) ColorMapGAN: Our main contribution is novel ColorMapGAN, which can generate fake training images that are semantically exactly the same as the original training images (i.e., location and shape of the objects such as roads, building, trees, etc. are exactly the same in the fake and in the original training images) and that are visually indistinguishable from the test images (i.e., the objects such as trees and buildings in the fake training images and the test images have similar spectral distributions). To do this, ColorMapGAN transforms the colors of the training images into the colors of the test images without doing any structural changes on the objects of the training images.

2) Higher accuracy: In our experiments, we perform city to city domain adaptation. We utilize the fake training images generated by our approach and by CycleGAN [26], UNIT [27], MUNIT [28], DRIT [29], histogram matching [47], and gray world algorithm [61] to fine-tune the model trained from the original training images. We also compare our approach with AdaptSegNet [12], which aims at adapting the classifier to the test data directly. Despite its architecturally simple design, our framework enables to generate predicted maps with much higher accuracy than the others.

3) Lower complexity: Unlike the existing GANs in the literature, our generator does not perform any convolution or pooling operations. It transforms the colors of the training images with only one element-wise matrix multiplication and one matrix addition. Because our generator is architecturally substantially simpler compared to the existing GANs, the training time of ColorMapGAN is significantly lower than the other learning based approaches. In our experiments, we compare our approach with CycleGAN [26], UNIT [27], MUNIT [28], DRIT [29], histogram matching [47], and gray world algorithm [61] in terms of running times.

II Generative Adversarial Neural Networks

In machine learning, we can divide the models trained in a supervised setting into two groups: discriminative and generative models. In the field of image analysis, the discriminative models are usually trained to learn a mapping from a high dimensional input to class labels as in image categorization and segmentation problems. On the other hand, the generative models aim to estimate the distribution of the data samples so that new samples could be drawn from the estimation. In 2014, Goodfellow et al. proposed the generative adversarial networks (GANs) [62], which is a novel approach to train a generative model.

GANs usually comprise a generative model $G$ and a discriminative model $D$. The goal of $G$ is to estimate the distribution of the real data and to output fake data from the estimation. $G$ takes a random noise $z$ as input, and represents a mapping to data space $G(z)$. We denote the distribution of the real data $x$ by $p(x)$ and a prior on input noise variables by $p(z)$. Let us assume that the real data $x$ and the fake data $G(z)$ are indicated by 1 and 0, respectively. $D$ outputs a scalar between 0 and 1, and aims to maximize the probability of labeling $x$ and $G(z)$ correctly. In other words, the goal of $D$ is to discriminate between the real and the fake data. The objective function for $D$ that is maximized during training is described as:

[TABLE]

where $\mathbb{E}$ is the expected value. $G$ is simultaneously trained to minimize the objective function defined as:

[TABLE]

As a result, the minimax game played between G and D could be formulated as:

[TABLE]

Once $D$ and $G$ are simultaneously trained for a sufficiently long time, $D$ becomes good at discriminating the fake and the real data, and $G$ becomes better at generating fake data that are indistinguishable from the real data.

Although GAN works well with shallow multi-layer perceptrons, they suffer from instability problems during training when a more complex network is used. Several approaches have been presented to address the instability issues. In DCGAN [63], instead of multilayer perceptrons, deep convolutional networks are used in both $G$ and $D$, and certain architectural constraints are introduced for a more stable training. In WGAN [64], rather than the logarithms in Eq. (3), Wasserstein distance is used to compute distance between the distributions, and gradient clipping is applied in the training stage. WGAN-GP[65] is an extension of WGAN, where a gradient penalty is performed to solve the limitations of the gradient clipping. LSGAN [66] proves that adopting the least squares loss function in Eq. (3) allows more stabilized training.

Finally, the original GANs were extended to conditional GANs in [67], where instead of generating the fake data from noise $z$, both $G$ and $D$ are conditioned on some extra information $y$. $y$ can be class labels or data from other modalities. In this architecture, $G$ learns a mapping from combination of $z$ and $y$ to the data space. If we consider $y$ as the source data and $x$ as the target data, $G$ aims to learn a mapping from source domain to target domain. Inspired from this idea, conditional GANs have been used for several I2I works [60, 26].

III Methodology

III-A The Overall Framework

Fig. 2 depicts the overall framework that consists of 4 steps as follows:

Training the initial classifier: We train a U-net on the original training data. 2. 2.

ColorMapGAN: We generate fake training images that are semantically exactly the same as the original training images, but visually as similar as possible to the test images using the proposed ColorMapGAN. 3. 3.

Fine-tuning: We fine-tune the model obtained in step 1 using the fake training images and the ground-truth for the original training images. 4. 4.

Classification: Finally, we classify the test images.

We use a slighly modified version of U-net [1] as the classifier. We replace rectified linear activation units (ReLU) by leaky rectified linear activation units (Leaky-ReLU) for a better performance [68]. We also remove the batch normalization operation in each layer, since it uses the memory inefficiently. In the framework, the steps 1, 3, and 4 are self-explanatory, whereas step 2 needs further explanation.

III-B ColorMapGAN

1) Generator: The novelty of the proposed ColorMapGAN is in the architecturally simple but powerful design of its generator.

We denote a set of training image patches by $X=\{x_{1},x_{2},\ldots,x_{N}\}$ and a set of test image patches by $Y=\{y_{1},y_{2},\ldots,y_{M}\}$. $G(X)$ corresponds to the set of fake training patches generated by $G$. The goal of $G$ is to generate $G(X)$, whose spectral distribution is as similar as possible to the distribution of $Y$, while keeping $G(X)$ and $X$ semantically exactly the same. Contrary to the existing GANs in the literature, we do not use convolutional or pooling layers in $G$ to preserve the exact semantics of $X$ in $G(X)$.

Let us assume that $X$ and $Y$ are composed of 8 bit images comprising red, green, and blue channels. We denote by $R=\{0,1,\ldots,255\}$, $G=\{0,1,\ldots,255\}$, and $B=\{0,1,\ldots 255\}$, the values each color band of the pixels can take. We denote all the possible $16,777,216$ ($256\times 256\times 256$) colors by $RGB$, which is defined as:

[TABLE]

where $\bigtimes$ stands for the Cartesian product. In order to transform $RGB$ to another color matrix $R^{\prime}G^{\prime}B^{\prime}$, we use a scale $W$ and a shift $K$ matrices with the same size as $RGB$. $R^{\prime}G^{\prime}B^{\prime}$ could be computed as:

[TABLE]

where $\circ$ is the element-wise product. The only learnable parameters of our $G$ are $W$ and $K$. Before the training starts, we initialize $W$ with ones and $K$ with zeros. Hence, at the end of the first iteration, the input and the output of $G$ are exactly the same.

The main bottleneck for computing Eq. (5) is the computational complexity. Since each of $RGB$, $W$, and $K$ matrices has more than 50 millions of elements (256 $\times$ 256 $\times$ 256 $\times$ 3), it is not feasible to perform the operation defined in Eq. (5) on a GPU. However, the number of colors in a training image patch is much lower than the number of all the possible colors. Therefore, it is sufficient to update only the elements of $W$ and $K$ which transform the colors that are available in the training patch. To do this, we use an index vector $I$ that is defined as:

[TABLE]

where $r$, $g$, $b$ are red, green, blue values of all the pixels in the training patch. After the elements of $I$ are found, we normalize and center each $x_{i}\in X$ and $y_{j}\in Y$ by first dividing by $127.5$ and then subtracting $1$ so that each color channel ranges between $-1$ and $1$. We then partially update $R^{\prime}G^{\prime}B^{\prime}$ as:

[TABLE]

where $[\cdot]$ operation corresponds to retrieving the rows of an arbitrary 2D matrix indexed by the given vector. Then, in $R^{\prime}G^{\prime}B^{\prime}[I]$, we replace the elements that are bigger than 1 and smaller than -1 by 1 and -1, respectively. To range all the values in $R^{\prime}G^{\prime}B^{\prime}[I]$ between 0 and 255, we then use the denormalization function $DN$ that is defined as:

[TABLE]

where $p$ is a 2D input matrix. The final output of $G$ can be obtained by reshaping $DN(R^{\prime}G^{\prime}B^{\prime}[I])$ back to the shape of the input patch. In each training iteration, we update only $W[I]$ and $K[I]$.

Fig. 3 illustrates the overall flowchart for the feedforward pass of $G$.

*2) Discriminator: * The discriminator in ColorMapGAN is architecturally the same as the discriminator of CycleGAN[26] (see Fig. 4). Instead of outputting a single scalar for the whole image patch to determine if the patch is real or fake, this discriminator generates a two dimensional matrix. Each element of the matrix is used to locally determine whether the input patch is real or fake. We then take average of all the elements of the matrix to yield a final value.

As we mention in Sec. II, GANs suffer from the instability issues; therefore, other objective functions have been proposed as an alternative to Eqs. 1, 2, and 3. We prefer to use the functions presented in LSGAN [66]. Thus, for $D$, the objective function that is minimized during training becomes:

[TABLE]

The objective function for $G$ is defined as:

[TABLE]

We train $D$ and $G$ simultaneously by minimizing the Eqs. 9 and 10.

IV Experiments

IV-A Methods Used for Comparison

U-net[1]: We simply train a U-net from the training images and segment the test images without performing any type of domain adaptation techniques.

CycleGAN[26]: In this methodology, in addition to $G$, there is one more generator $F$. $G$ is used to learn a mapping from training images to test images, whereas $F$ learns a mapping from test images to training images. The methodology requires that the mapping from training images to test images and another consecutive mapping from test images to training images reproduce the original training images. This constraint is enforced by minimizing the L1 norm between $F(G(X))$ and $X$, and between $G(F(Y))$ and $Y$.

UNIT[27]: The generator of UNIT has two encoders $E_{x}$ and $E_{y}$ and two decoders $G_{x}$ and $G_{y}$. The encoders are used to embed $X$ and $Y$ to a common space. The fake images are generated by $G_{x}(E_{y}(Y))$ and $G_{y}(F_{x}(X))$. In an ideal transformation, $X$ and $G_{x}(E_{y}(Y))$, and $Y$ and $G_{y}(E_{x}(X))$ have similar statistics.

MUNIT[28]: It decomposes the images from both domains into content and style codes. To generate fake images, the content code of one domain and the style code of another domain are combined. AdaIN [69] is utilized for the combination.

DRIT[29]: Methodologically DRIT is almost the same as MUNIT. The only difference is that the content code of one domain and the style code of another domain are combined by concatenating them.

Histogram matching[47]: For each color channel, we match the histogram of the training images with the histogram of the test images to correct the spectral shift between the images.

Gray world[61]: It is one of the color constancy algorithms [50]. This algorithm assumes that the average color of the image should be natural gray, and any deviation from gray is caused by the illuminant. This assumption is used to remove the effect of the illuminant. We use gray world algorithm to standardize the training and the test data.

AdaptSegNet single[12]: It aims at training a domain invariant network that performs well in segmenting both training and test images. To do this, the classifier generates predicted maps from the training and the test images, and the discriminator forces the predicted maps for the test images to look like the predictions for the training images. In the original paper, DeepLab v2 [70] is used as the classifier. However, Atrous Spatial Pyramid Pooling (ASPP) in this network reduces the segmentation performance on satellite images significantly, especially when the image contains objects covering a small area. Hence, we remove ASPP from the network and upsample the final classification layer directly to get the predicted map with exactly the same size as the training images.

AdaptSegNet multiple[12]: In the same paper, in addition to aligning the final predictions for the training and test images, the experimental results with 2 classification layers and 2 discriminators are also presented. We compare our method with this strategy as well.

To make a fair comparison between ColorMapGAN and CycleGAN, UNIT, MUNIT, DRIT, histogram matching, and gray world, we replace the step 2 in our framework with these algorithms. For gray world, we also modify the step 4. Instead of segmenting the original test image, we segment the test image in which the illuminant effect is removed.

IV-B Experimental Setup

To conduct our experiments, we use the Luxcarta data set containing Pléiades images acquired over 4 cities in Europe: Bad Ischl (Austria), Villach (Austria), Béziers (France), and Roanne (France). The images are converted to 8 bit, and their spatial resolution is reduced to 1m by the data set providers. The images in the data set contain red, green, and blue channels. The full annotations for building, road, and tree classes are provided. An example close-up from each city is shown in Fig. 5. We split the cities into two pairs, where the first pair consists of Bad Ischl and Villach, and the second pair comprises Béziers and Roanne. To make the experimental setup suitable for the unsupervised domain adaptation problem, when we split the cities into pairs, we pay attention that radiometry of both cities in each pair is as different as possible, and the objects belonging to the same class (e.g., building) have similar structural characteristics. For instance, buildings in Béziers and Roanne are densely grouped and have mostly rectangular shape, whereas buildings in Bad Ischl and Villach are more sparsely distributed and mostly square-like shaped. However, as can be seen in Fig. 5, there is a large domain shift between spectral bands of the cities in each pair. Our final assumption is that we have access to annotations of only the training city. For instance, when we classify Roanne in pair 2, we suppose that its annotations are not available; only the ground-truth of Béziers is accessible. Similarly, to classify Béziers, we assume that the ground-truth of only Roanne is accessible.

In the preprocessing step, we split each satellite image into $256\times 256$ training patches with an overlap of $32$ pixels between neighboring patches. Table I reports for each city the number of patches, the total area covered, and the class frequencies. For the quantitative performance assessment, we use Intersection over Union (IoU)[71] as the evaluation metric.

IV-C Training Details

In the first step of the framework, we train a U-net with Adam optimizer, where the learning rate is 0.0001, exponential decay rate for the first and the second moment estimates are 0.9 and 0.999. In each iteration, we randomly sample a batch of 32 training patches. We apply online data augmentation with random horizontal/vertical flips and 0/90/180/270 degrees of rotations. We use sigmoid cross entropy as the loss function and ignore the background class while computing the loss. We optimize the network for 2,500 iterations.

In the stage of generating fake training images, in each training iteration of ColorMapGAN, we randomly sample only one patch from the training city and one patch from the test city. We use Adam optimizer to update both the generator and the discriminator. Since the generator is architecturally much simpler than the discriminator, we prefer to optimize it with a larger learning rate. For the generator, the learning rate is 0.0005, whereas we set it to 0.0001 for the discriminator. We train ColorMapGAN for 8,000 iterations, since we verify by visual inspection that visually appealing results are obtained for this number of iterations. We fine-tune the previously trained network on the fake training images for 750 iterations. For all the compared methods, we use default parameters that are specified in the related papers.

IV-D Results

Tables II and III depict IoU values for each method on the test set. To provide as reliable as possible results, we repeat the step 3 in our framework 20 times for each method that generates fake data, and report the average IoU values in the tables. Re-running the step 3 of the framework 20 times is feasible, since we fine-tune the classifier for only 750 iterations in this step. However, AdaptSegNet tries to adapt the classifier to the test data directly. Hence, we need to train a classifier from scratch every time when we repeat the experiment. For this reason, for AdaptSegNet Single and Multi, we show the average IoU values for only 3 runs. Figs. 14 to 17 depict the predictions. Because we run the experiments for each method multiple times, we illustrate the predictions that are obtained by majority voting. In these figures, CycleGAN, UNIT, MUNIT, DRIT, Gray world, Hist. match., and ColorMapGAN represent the results for our framework with these method. We do not add ”our framework with” statement in the captions for the sake of simplicity.

Figs. 6 to 9 illustrate some parts of the original training images from the pairs and the fake training images generated by CycleGAN, UNIT, MUNIT, DRIT, gray world, histogram matching, and ColorMapGAN. From the images, we can clearly see that MUNIT and DRIT spoil the semantic identity of the images completely. For instance, the structures indicated by yellow and green rectangles in the original images are either replaced by other objects or distorted in the fake images. In some cases, UNIT seems to generate better results. For instance, the fake cities generated by UNIT in pair 2 are semantically relatively consistent with the original images, and style-wise similar to the test cities. On the other hand, the fake cities in pair 1 have plenty of artificial objects that do not exist in the original city. For UNIT, MUNIT, and DRIT, since the fake images and the ground-truth for the original images usually do not match, the network learns wrong information in the step 3 of our framework. Another problem with these approaches is that we observe tiling effect in the fake images. Especially in the fake Villach images, the transition between the patches is not continuous. The reason is that since it is impossible to fit the large satellite images to GPU directly when generating fake images; we need to generate fake data from each patch and combine them to get the entire fake city. However, these approaches mostly generate irrelevant output for the neighboring patches. As a result, the proposed framework with these methods perform poorly on the test data, as confirmed by Tables II, III and Figs. 14 to 17.

As can be seen in Figs. 6 to 9, the spectral difference between the training and the test images can be reduced by standardizing them using the gray world algorithm. However, between the fake images in each city pair, we still observe a spectral shift; it is not completely corrected. In consequence, the performance of gray world algorithm is mostly better than UNIT, MUNIT, and DRIT, but it is not as good as desired.

The network architecture used in AdaptSegNet single is very deep; therefore, aligning only the outputs of the final classification layer for the training and the test images does not yield a good performance. However, if the alignment is performed in multiple layers, a better performance could be obtained, especially when the objects of interests cover a large area such as forests. For instance, most of the trees in Villach are located in forests areas. For tree class on this city, IoU value for AdaptSegNet multi is 56.08$\%$, which is slightly lower than the performance of our framework. However, the performance of AdaptSegNet is not satisfactory in segmenting small objects such as building and thin objects like roads.

At the first sight, histogram matching seems to be working well; semantic structures of the training city are well preserved in the fake training city, and the style of the test city is perfectly transferred to the fake training city. However, it is noticeable in Tables II and III that the quantitative results for this approach are very poor most of the time. Besides, the fake cities generated by CycleGAN and ColorMapGAN in pair 1 look similar. However, still there exists a large gap between the performance of the framework with ColorMapGAN and with CycleGAN. When segmenting Villach, for road class, the IoU of CycleGAN is around 9$\%$ lower than of ColorMapGAN. Similarly, ColorMapGAN outperforms CycleGAN by 6$\%$ for building class, when segmenting Bad Ischl. On the other hand, CycleGAN outperforms ColorMapGAN for tree class. To better understand the reasons for this performance difference, the comparisons between ColorMapGAN and CycleGAN, and ColorMapGAN and histogram matching need further analysis.

ColorMapGAN vs CycleGAN [26]: First of all, it is worth noting that the performance of CycleGAN is unstable. The proposed framework with CycleGAN performs unsatisfactorily on pair 2 because of its semantically inconsistent outputs with the original images. As highlighted by yellow rectangles in Figs. 8 and 9, CycleGAN removes some objects that exist in the original cities. There are several reasons why it performs worse than ColorMapGAN on pair 1 for building and road classes. Firstly, the resolution of its output is lower than the resolution of the original city and the output of ColorMapGAN. Fig. 10 illustrates a closeup from Villach and the corresponding fake images generated by ColorMapGAN and CycleGAN. The resolution difference between the fake images can easily be noticed in the outlined areas by cyan rectangles. Learning from blurrier data obviously deteriorates the performance. Secondly, the output of CycleGAN has some artifacts as shown in the same figure by a yellow rectangle. The generator of ColorMapGAN does not have convolution, pooling, etc. operations; therefore, we do not observe such artifacts in the outputs of ColorMapGAN. Finally, since we generate fake cities patch by patch because of memory constraints, there exists a spectral difference between some of the neighboring patches in the fake images generated by CycleGAN (see the green rectangle in Fig. 6b). This difference leads the network to exhibit a lower performance. ColorMapGAN does not have this problem, because it maps each color to another one. The pixels having the same color are mapped to another exactly the same color, irrespective of their locations. Therefore, the neighboring patches are spectrally consistent, and there is no tiling effect between them. One drawback of ColorMapGAN is that it seems to be slightly smoothing out trees, which results in the patterns on trees disappearing slightly (see the green rectangle in Fig. 10c). This is probably why the framework with CycleGAN outperforms the framework with ColorMapGAN on pair 1 for tree class.

ColorMapGAN vs Histogram matching [46]: The main problem of histogram matching is that it does not take into account the contextual information, it only tries to match the histogram of the whole training city with the histogram of the whole test city. Conversely, the discriminator of ColorMapGAN extracts high level features from the output of the generator and the test data to decide which one is real and which one is fake. In other words, the generator generates a fake training city in a way that its high level features align with the high level features extracted from the test city. For this reason, the proposed framework with ColorMapGAN yields substantially improved results. As shown in Fig. 11 by yellow rectangles, when generating fake Bad Ischl, histogram matching converts some gray rooftops to cyan ones, whereas ColorMapGAN keeps them gray. Similarly, in the same figure, we observe that the buildings highlighted by green rectangles have dark violet rooftops in the output of histogram matching, and black rooftops in the image generated by ColorMapGAN. In Fig. 7a, we can see that there is no building having a cyan or dark violet rooftop in Villach, but there exists lots of buildings with gray or black rooftops. If the generator of ColorMapGAN generated cyan or dark violet colored rooftops, the discriminator would easily understand that these buildings were fake. For this reason, such buildings do not appear in the output of ColorMapGAN. Similarly, in the process of generating fake Roanne, histogram matching algorithm generates some reddish roads, as shown in Fig. 12 by a yellow rectangle. In the same figure, we see that ColorMapGAN outputs gray roads. Moreover, ColorMapGAN generates buildings having brownish colored rooftops, which are probably more representative for the buildings in Béziers than the buildings with red rooftops generated by histogram matching. As discussed in Sec. I, CNNs are extremely sensitive to the training data. A small domain shift between the training and the test data may affect the results significantly. Furthermore, in Fig. 13, we depict color histograms of the buildings in Roanne, fake Roanne generated by histogram matching and ColorMapGAN, and Béziers. Since Roanne and Béziers are two different cities, we cannot expect the histograms of an ideal fake Roanne and Béziers to be exactly the same. However, we expect them to resemble each other. Although histogram matching algorithm tries to match the histogram of the training city with the histogram of the test city, there exists a large difference between the class-wise histograms of the fake training and the test cities. As can be seen in Fig. 13b, there is a large deviation between some of the neighboring bins in the histogram. In contrast, color histograms of the buildings in the fake Roanne generated by ColorMapGAN are more similar to the histograms of the buildings in Béziers. For histogram matching, we observe the same issue for the other classes as well, but we do not include the histograms of the other classes because of the lack of space.

IV-E Running Times

The proposed framework, ColorMapGAN, CycleGAN, UNIT, MUNIT, and DRIT were implemented in Tensorflow111https://www.tensorflow.org. We conducted all the experiments on an Nvidia Geforce GTX1080 Ti GPU with 11 GB of RAM. Table IV reports the running times for training ColorMapGAN and the other learning based compared methods for 1 iteration. The table demonstrates that the training time for ColorMapGAN is significantly shorter than the other learning based approaches. Let us also remark again that we optimize ColorMapGAN for only 8,000 iterations. In other words, we need only about 6.5 minutes to train it. On the other hand, the other learning based approaches with the default parameters requires long hours to train. It is also notable that non-learning based approaches generate an output in a substantially shorter time. The execution time for gray world algorithm is almost instant, and histogram matching needs less than half a minute. However, the quality of the results for non-learning based methods is deficient.

V Concluding Remarks

In this work, we presented a novel framework to generate high quality maps from satellite images even when there exists a large domain shift between spectral bands of the training and the test images. We validated our approach on two city pairs, where we performed a city to city domain adaptation in each pair. Our experimental results demonstrated that the proposed framework exhibits a much better performance than the existing approaches. We also showed that the proposed ColorMapGAN generates fake images in a significantly shorter time than some of the competitive unpaired image to image translation methods in the computer vision community.

A possible future direction could be to investigate a more difficult domain adaptation problem, where in addition to the large spectral difference, shapes of the objects such as buildings in the training and test images are considerably different.

Acknowledgment

The authors would like to thank ACRI-ST and CNES for initializing and funding this study. The authors also thank Alain Giros and Sébastien Clerc for fruitful discussions.