Bridging the Domain Gap for Ground-to-Aerial Image Matching

TL;DR

This paper introduces a novel approach using conditional GANs to synthesize aerial images from ground panoramas, effectively reducing domain gap and improving cross-view image matching accuracy.

Contribution

The method leverages generative models and feature fusion techniques to enhance cross-view image matching, outperforming existing state-of-the-art approaches.

Findings

Significantly better top-1 and top-1% retrieval accuracies on CVUSA dataset.

Effective synthesis of aerial images from ground panoramas improves domain alignment.

Generalization demonstrated on a new urban landscape dataset.

Abstract

The visual entities in cross-view images exhibit drastic domain changes due to the difference in viewpoints each set of images is captured from. Existing state-of-the-art methods address the problem by learning view-invariant descriptors for the images. We propose a novel method for solving this task by exploiting the generative powers of conditional GANs to synthesize an aerial representation of a ground level panorama and use it to minimize the domain gap between the two views. The synthesized image being from the same view as the target image helps the network to preserve important cues in aerial images following our Joint Feature Learning approach. Our Feature Fusion method combines the complementary features from a synthesized aerial image with the corresponding ground features to obtain a robust query representation. In addition, multi-scale feature aggregation preserves image representations at different feature scales useful for solving this complex task. Experimental results show that our proposed approach performs significantly better than the state-of-the-art methods on the challenging CVUSA dataset in terms of top-1 and top-1% retrieval accuracies. Furthermore, to evaluate the generalization of our method on urban landscapes, we collected a new cross-view localization dataset with geo-reference information.

Tables

Table 1. Table 1: Comparison of Top-1, Top-10 and Top-1% recall for the baselines and the proposed approaches (first panel) and with previous methods (second panel) on CVUSA Dataset [ 46 ] .

Method	Top-1	Top-10	Top-1%
Two-stream baseline ($I_{a^{\prime }}$, $I_a$)	$10.23\%$	$35.10\%$	$72.58\%$
Two-stream baseline ($I_g$, $I_a$)	18.45%	$48.98\%$	82.94%
Joint Feat. Learning ($I_{a^{\prime }}$, $I_a$)	14.31%	48.75%	86.47%
Joint Feat. Learning ($I_g$, $I_a$)	29.75%	66.34%	92.09%
Feature Fusion	48.75%	81.27%	95.98%
Workman et al. [41]	-	-	34.3%
Zhai et al. [46]	-	-	43.2%
Vo and Hays [39]	-	-	$63.7\%$
CVM-Net-I [18]	22.53%	$63.28\%$	91.4%
CVM-Net-II [18]	11.18%	43.51%	87.2%

Table 2. Table 2: Ablation Study on CVUSA Dataset [ 46 ] . The reported numbers are the retrieval accuracies for Feature Fusion network for specified ablation criteria.

Ablation Criteria	Top-1	Top-10	Top-1%
Single Scale Features	$8.01$ %	$32.62$ %	$74.41$%
Global Avg. Pooling (GAP)	$16.13\%$	$51.72\%$	$87.68\%$
Weight Sharing	$29.94\%$	$68.24\%$	$93.42\%$
Multi-scale Features
+ No GAP + No Wt. Sharing	48.75%	81.27%	95.98%

Table 3. Table 3: Top-1 retrieval accuracy on Orlando-Pittsburgh Dataset.

Two-stream ($I_g$, $I_a$)	Joint Feat. Learning	Feature Fusion
$30.61\%$	$38.36\%$	$45.57\%$

Table 4. Table 4: Image matching performance in terms of Top-1, Top-10 and Top-1% recall on CVUSA Dataset [ 46 ] for aerial-to-ground matching.

Method	Top-1	Top-10	Top-1%
Two-stream baseline ($I_{g^{\prime }}$, $I_g$)	$15.04\%$	$37.31\%$	$67.99\%$
Two-stream baseline ($I_a$, $I_g$)	16.99%	$47.06\%$	82.11%
Joint Feat. Learning ($I_{g^{\prime }}$, $I_g$)	16.46%	50.26%	86.26%
Joint Feat. Learning ($I_a$, $I_g$)	27.39%	65.29%	91.46%
Feature Fusion	44.99%	79.37%	95.66%

Equations

$$L_{triplet}=max(0,m+d_p-d_n),$$

$$L_{soft}=ln(1+e^d),$$

$$L_{weighted}=ln(1+e^{\alpha d}).$$

$$L_{joint}={\lambda }_1{L}_{weighted}(I_g,I_a)+{\lambda }_2{L}_{weighted}(I_{a^{\prime }},I_a).$$

Full text

1 Introduction

Estimating the geo-location of an image has been tackled as an image-matching task, where the query image is compared against a database of reference images with known locations. Traditionally, the matching has been conducted between images taken from the same view, primarily street-view [14, 35, 44], which have a high degree of visual similarity in terms of scene contents. Since these ground level reference images are typically concentrated around urban areas with more human accessibility, the applicability of the method is limited to those regions. With the availability of aerial images from Google maps, Bing maps, etc. that cover the earth surface densely, researchers have lately explored the prospect of cross-view image matching [18, 24, 41], where the query ground image is matched against aerial images. This comes with additional challenges due to variation in viewpoints between the ground and aerial images, which capture the same scene differently in two views. This motivates us to explore transforming the query street-view image into aerial view, so that the transformed image has scene representations similar to the images it is matched against.

The recent success of Generative Adversarial Networks (GANs) [12] in synthesizing realistic images from randomly sampled noise vectors [30] or conditional variables such as text [31, 47], images [19, 32], labels [28], etc. has inspired us to frame the problem as viewpoint translation followed by feature matching. Moreover, GANs have been used for domain transfer problems as in [20, 48] to learn the mapping between different domain representations. Recent cross-view synthesis works by [32, 33, 9, 49] are successful in transforming the images between aerial and street views. In this paper, we address the following problem: given a ground-level panorama retrieve matching aerial images. In order to solve this problem, we take a next step to synthesize aerial images from ground-level panorama and use them for image retrieval.

The complexity of the cross-view image synthesis problem and its challenges are well-known. Thus, the synthesized images cannot be relied on to completely replace the query ground-level image to solve the matching task. Therefore, we propose a framework as shown in Figure 1 to incorporate the synthesized image into the matching pipeline as auxiliary information in order to bridge the existing domain gap between aerial and ground views. We attempt to learn representations for aerial reference images that are similar to their corresponding ground level images, as well as the synthesized aerial images. Since the synthesized aerial images are transformed representations of street-view (ground) images, we expect them to contain representative features. By learning representations in this manner, the synthesized aerial images force the network to minimize the distance between feature representations of aerial images and street-view images. Additionally, we hypothesize that some features of aerial images are better learned by considering synthesized aerial images rather than street-view images. Thus, the joint training of these image triads (ground, synthesize aerial from ground, and corresponding real aerial) will help the aerial stream retain important cues that would have otherwise been lost in cross-view training. We fuse the learned complementary feature representations of synthesized images with query image features to obtain a robust representation that we use for our image matching task.

The features extracted at different layers of deep neural networks capture varying levels of semantic information of the input image. For the image matching task, which is considerably more challenging than a standard classification problem, we exploit the inherent multi-scale pyramidal structure of features at multiple layers of deep neural networks and aggregate them to obtain a better image representation.

In summary, this paper makes the following contributions. We propose a novel approach to leverage aerial images synthesized using GANs to extract complementary features for cross-view image matching. We incorporate the edgemaps, in addition to semantic segmentation which is typically used, together with the input images to improve the cross-view synthesis by providing cues on object shapes and boundaries to the network. The synthesized images bridge the domain gap between cross-view images. The joint training of image triads using auxiliary loss helps improve the network training. The proposed feature fusion strategy demonstrates the capabilities of GANs for constructive training and complementary feature learning. Lastly, we show that aggregating features from multiple convolutional layers at different resolutions greatly helps preserve coarse to fine latent representations necessary for complex cross-view matching task. Our extensive experiments show that the proposed joint feature learning method outperforms the state-of-the-art methods on CVUSA dataset [46] and with feature fusion, we obtain significant improvements on top-1 and top-10 retrieval accuracies.

2 Related Works

2.1 Domain Transfer and GANs

GANs are very popular in domain transfer tasks. In the works reported in [19, 48, 20, 42, 10], image mapping between two domains; source and target domains is learnt. Augmented CycleGAN [2], StarGAN [7] have explored many-to-many cross-domain mappings.

Cross-view relations have been explored in [46, 32, 11] with more challenging settings of aerial and ground views, where there is minimum semantic and viewpoint overlap between the objects in the images. Cross-view image synthesis between these contrasting domains has attracted wide interests lately [32, 33, 9, 49] with the popularity of GANs; these works have been successful in image translation between aerial and ground-level cropped (single camera) images. Zhai et al. [46] explored the possibility of synthesizing ground-level panorama from ground semantic layouts wherein the layouts were predicted from the semantic maps of the aerial images. Here, we directly transform the ground level panorama to aerial view and use them for cross-view image matching task.

2.2 Multi-scale Feature Aggregation

Features at different layers of deep neural networks are essentially the multi-resolution features of the same image. Abundance of literature has explored features at multiple scales [17, 29, 34, 26, 22] for applications like key-point detection, human pose estimation, semantic segmentation. FPN [25], HyperNet [21], ION [5] explored multi-scale features for object detection. Earlier, Hypercolumns [13] were created from multi-layer features and used for object segmentation and localization. Building upon this work, we also aggregate the features at multiple scales to efficiently obtain robust representation of the images.

2.3 Image Geolocalization

Image geolocalization has been tackled as an image matching task [3, 15, 43] in computer vision community. Early works in geolocalization [44, 38, 35, 45] matched images in the same view; a query street-view image is compared against the reference street-view images using hand-crafted features. Hays et al. [14] proposed a data-driven approach to estimate the distribution over geographical location from a single image.

Cross-view matching has been explored by several recent works [24, 36, 18, 40, 39] using both hand-crafted features as well as deep networks. Bansal et al. [4] explored facade matching. Tian et al. [37] matched building features in oblique views. Recent work by [18] exploit the NetVLAD [3] to obtain view-invariant descriptors for cross-view pairs and use them for matching.

In this work, we exploit the synthesized aerial images as complementary source of information for better scene understanding of street-view images to solve cross-view matching task, rather than just learning view-invariant features as in the previous approaches.

3 Method

We propose a novel method to bridge the domain gap between street-view and aerial images by leveraging the synthesized aerial images using GANs. We learn the representations of synthesized aerial images jointly with ground and aerial image representations. Additionally, we fuse the complementary representations of ground images with the representations of their corresponding synthesized aerial images to learn robust query representations of ground images. Also, we exploit the edgemaps of input images to provide GANs with the notion of object shapes and boundaries and facilitate the cross-view image synthesis.

The organization of the rest of this section is as follows. In the next subsection, we briefly describe how GANs are used for cross-view image synthesis, followed by joint feature learning, and finally feature fusion is described.

3.1 Cross-View Image Synthesis

We adopt X-Fork generator architecture of [32] to train the GAN for cross-view image synthesis. The X-Fork is a multi-task learning architecture that synthesizes cross-view image as well as semantic segmentation map. We make the following modifications to the X-Fork architecture. Since our input is panorama (rectangular in shape), the feature maps at the bottleneck are also rectangular (1 $\times$ 4). We reshape the features into squares (2 $\times$ 2), and then apply multiple upconvolution operations to generate 512 $\times$ 512 resolution aerial images. Next, we exploit the edgemaps of input images that outline the objects present in the images. We employ Canny Edge Detection [6] to obtain the edgemaps of the inputs. The edgemap is stacked together with the panorama, along the channels to create a 4-channel input; 3 channels for RGB image and 1 channel for edgemap. The output is an RGB image and its segmentation map in aerial view. We utilize the synthesized aerial images in joint feature learning experiments.

3.2 Joint Feature Learning

We propose to learn the representations for image triads: query ground panorama, $I_{g}$, synthesized aerial image, $I_{a^{\prime}}$ from ground panorama and aerial image $I_{a}$ jointly, so that the synthesized aerial image representations $f_{a^{\prime}}$ pushes the image representations $f_{g}$ and $f_{a}$ closer to each other.

The joint feature learning architecture is shown in Figure 2(a). The encoder blocks are shown in green (for ground image) and blue (for aerial images) triangles. Each encoder consists of deep convolutional architecture as described in subsection 4.2. We elegantly exploit the inherent multi-scale pyramidal structure of features at multiple layers of deep neural networks. We consider the features from the final three convolutional layers, conv_6, conv_7 and conv_8 layers. These features are aggregated and followed by a fully connected layer to obtain the feature representation for images in each view.

The encoders for aerial and street-view images do not share the weights. Since the cross-view images are captured from different viewpoints, the visual entities exhibit drastic domain changes. The two encoders operate on these sets of diverse images, so it is understandable that the weight sharing is not a good choice. On the other hand, the encoders for $I_{a^{\prime}}$ and $I_{a}$ share the weights, since both images represent the aerial domain. This way, the aerial encoders learn weights suitable for the synthesized image $I_{a^{\prime}}$ as well as the real image $I_{a}$. Thus, $f_{a^{\prime}}$ effectively forces the features $f_{a}$ to be closer to $f_{g}$ and bridges the domain gap between the two views. This is possible because the transformed image $I_{a^{\prime}}$ captures representations of $I_{g}$ which are easier for the network to learn from $I_{a^{\prime}}$ than it would be when learning directly from $I_{g}$.

This strategy leverages the synthesized images at training time, but does not require them during the testing. The auxiliary loss between $I_{a^{\prime}}$ and $I_{a}$ influences the aerial image encoder to learn representations for aerial images by considering the synthesized aerial image. We train our network jointly on these image triads ($I_{g}$, $I_{a^{\prime}}$ and $I_{a}$) using weighted soft-margin ranking loss [18], which is explained next.

3.2.1 Weighted Soft-margin Triplet Loss

Consider a feature embedding $f_{g}$ of ground-level image, $f_{a-pos}$ of the corresponding matching aerial image and a non-matching aerial image feature $f_{a-neg}$. The triplet loss [16] aims to bring the matching feature $f_{a-pos}$ closer to $f_{g}$ while at the same time pushes $f_{a-neg}$ away. Here, if $d_{p}$ is the Euclidean distance between positive samples ($f_{g}$, $f_{a-pos}$) and $d_{n}$ is the Euclidean distance between negative/non-matching samples ($f_{g}$, $f_{a-neg}$), we try to minimize $d_{p}$ as well as maximize $d_{n}$. The triplet loss is expressed as shown below:

[TABLE]

where, $m$ is a margin that specifies a minimum distance between non-matching pairs.

In order to avoid the necessity of explicitly deciding the margin for triplet loss, soft-margin triplet loss is popular and is expressed as given in Equation 2 below:

[TABLE]

where $d=d_{p}-d_{n}$.

In our work, we use the weighted soft margin triplet loss [18] as given in Equation 3:

[TABLE]

We use $\alpha$ = 10, which results in better convergence than $\alpha$ = 1.

We incorporate the auxiliary loss between the synthesized aerial images, $I_{a}^{\prime}$, and the real aerial images, $I_{a}$, along with the loss between the real aerial, $I_{a}$, and the ground images, $I_{g}$, for joint feature learning using the Equation 4 below:

[TABLE]

Here, $\lambda_{1}$ and $\lambda_{2}$ are balancing factors between the losses for ($I_{g}$, $I_{a}$) and ($I_{a^{\prime}}$, $I_{a}$) pairs respectively.

3.3 Feature Fusion

In the above method, the synthesized aerial image is used during the training only, for bridging the domain gap between the real aerial and ground view images; but is neglected during testing. Since the features of the synthesized image contain complementary information that assist in joint feature learning, we attempt to further exploit them. We fuse the ground image features $f_{g}$ with synthesized aerial image features $f_{a^{\prime}}$ and find a robust representation $f_{g^{*}}$ for the query ground image.

The fusion architecture is shown in Figure 2(b). We use the trained joint feature learning network as feature extractor for our feature fusion task. We first concatenate the features from ground query image with the features from synthesized aerial image. The concatenated features need to be refined to obtain a generalized representation for query image $f_{g^{*}}$. We achieve this by passing through a fully-connected layer in the upper stream. The features $f_{a}$ from the lower stream need to be optimized against the refined features from upper fully-connected layer. So, we add a fully-connected layer in the lower stream that learns the generalized representations, $f_{a^{*}}$, for the aerial images. During the testing, the fused feature representation $f_{g^{*}}$ for query image $I_{g}$ is compared against the representations $f_{a^{*}}$ for aerial images for image matching.

4 Experimental Setup

This section deals with the datasets we used and the experimental setups we followed in our work.

4.1 Datasets

We conduct experiments on CVUSA dataset [46] to compare our work with existing methods. We also collect a new dataset, OP dataset, from urban areas of Orlando and Pittsburgh with geo-information. The other benchmark dataset, GT-Crossview [39] doesn’t contain the ground level panorama, thus making it infeasible to synthesize meaningful aerial image. Also, the GT-Crossview dataset has aligned image pairs in training set, whereas unaligned image pairs in test set with no direction information, so the synthesized aerial images for test case will be randomly oriented relative to aerial images in the reference database, thus it is not possible to use this dataset in our framework.

**CVUSA: ** CVUSA is a benchmark dataset for cross-view image matching with 35,532 satellite and ground-panorama image pairs for training and 8,884 pairs for testing. Aerial images are $750\times 750$ and ground-panorama are $224\times 1232$ in resolutions. Sample images from this dataset are shown in Figure 3.

**Orlando-Pittsburgh (OP) dataset: ** We collect image pairs from two US cities, Orlando and Pittsburgh with ground-truth geo-locations. We call it Orlando-Pittsburgh (OP) dataset. The dataset covers urban areas of the cities, entirely different from the rural areas in CVUSA dataset. Figure 6 shows some example images of this dataset. The dataset contains 1,910 training and 722 testing pairs of aerial and ground-panorama images. The resolutions are $640\times 640$ for aerial images and $416\times 832$ for panoramas. Primary motivation to collect this dataset is to evaluate the generalization of the proposed methods in urban locations and to compute matching accuracy in terms of distance (meters); and the unavailability of such datasets publicly. Though small-scale, this dataset should be useful for future research in this direction.

4.2 Implementation Details

We present the implementation details of our cross-view synthesis network and the proposed image matching networks in this section.

**Cross-View Synthesis network: ** The generator of cross-view synthesis network, shown as Generator in Figures 1 and 2 has an encoder and two decoders, similar to the X-Fork architecture in [32]. The input to the encoder is a 4-channel image; 3-RGB channels and an edgemap, stacked together. The decoders generate cross-view image and its segmentation map, for a given input. The network consists of blocks of Convolution, Batch Normalization and Leaky ReLU layers. Convolutional kernels of size $4\times 4$ with a stride of 2 are used that downsamples the feature maps after each convolution, and to upsample the feature maps after each upconvolution operation. We reshape the features at bottleneck to adjust the feature shape and pass through the decoders. The six blocks of decoders share the weights whereas the final two blocks don’t. The discriminator network has similar architecture to the one used in [32]. We train the GAN end-to-end using Torch [8] implementation. The weights are initialized with a random Gaussian distribution with zero mean and 0.02 standard deviation.

**Joint Feature Learning network: ** Each stream (encoder) of joint feature learning network in Figure 2(a) consists of seven convolutional layers, each followed by ReLU activations. Dropouts are applied after the final three ReLU layers. The features after these dropouts are flattened and then concatenated to obtain multi-scale representation of the input image. This is followed by a fully-connected layer for dimensionality reduction to obtain 1,000-dimensional feature vector for each input. The two-stream baselines are trained from scratch with Xavier initialization. The joint feature learning network is initialized with weights from the two-stream network trained on ($I_{g}$, $I_{a}$) image pairs and the loss function is optimized as shown in Equation 4. We use $\lambda_{1}$ = 10 and $\lambda_{2}$ = 1, weighing more on the loss term for ($I_{g}$, $I_{a}$) pairs because of their superior performance over ($I_{a^{\prime}}$, $I_{a}$) in image matching as reported in Table 1 and objectively $I_{a}^{\prime}$ is used as an auxiliary information, only during the training in joint feature learning network.

Feature Fusion network The Feature Fusion network in Figure 2(b) has two fully-connected layers, one each for aerial and ground feature branches. The upper FC layer takes 2000-dimensional fused feature and translates it to a 1000-dimensional feature representation. The input to the lower FC layer is $f_{a}$ that is mapped to a 1000 dimensional feature representation. The FC layers are randomly initialized with a uniform distribution with zero mean and 0.005 standard deviation.

The two-stream baselines and the proposed joint feature learning and feature fusion networks are implemented using Tensorflow [1] with Adam optimizer (lr = $10^{-5}$) and dropout = 0.5. A batch size of B = 30 for experiments on two-stream networks and B = 24 for joint feature learning networks is used. Weighted soft-margin triplet loss is used for training in all the experiments. An exhaustive mini-batch strategy [39] is employed to maximize the number of triplets within each batch. For each image in a batch of B images, we have 1 positive pair and (B-1) negative pairs for each ground image, and (B-1) negative pairs for each aerial image. So, for B images, we have B positive pairs and 2 x B x (B-1) negative pairs. Further training is continued with in-batch hard negative mining; by training each positive pair against the most negative sample (i.e. smallest distance) in the batch. Code and dataset is publicly available 111https://github.com/kregmi/cross-view-image-matching.

In summary, GAN is first trained to generate the cross-view image $I_{a^{\prime}}$ for the ground panorama $I_{g}$. Next, the synthesized images are used for joint feature learning in our proposed method.

5 Results

We present an extensive analysis of our proposed method demonstrating the effectiveness of synthesized images for image retrieval to bridge the domain gap between the cross-view images. We also provide the comparison of our work with the state-of-the-art methods on the CVUSA dataset. Finally, we present an evaluation on geo-localization task on the OP dataset.

5.1 Evaluation Metric

The common metric for evaluation of image based matching task is to compute the recall accuracy. A matching is successful for a query street-view image if the correct match lies within a set of closest images in Euclidean distance of the representative features. We report top-1% accuracy for ease of comparison with previous works. We also report top-1 and top-10 recalls on CVUSA dataset.

5.2 Results of Our Approach

We evaluate our model variants in terms of retrieval accuracy on the CVUSA dataset [46]. The results are reported in Table 1 (first panel).

Baseline Comparison (first and second rows in Table 1 (first panel)): The two-stream networks trained employing image pairs ($I_{g}$, $I_{a}$) and ($I_{a^{\prime}}$, $I_{a}$), where first image in each tuple is the query, are the baselines. We observe that the synthesized image $I_{a^{\prime}}$ as a query performs quite well with 72.58% for top-1% recall but slightly lower than $I_{g}$ as query (82.94%). This means that the synthesized images capture fair amount of information from the ground panorama whereas they are not yet completely dependable for cross-view image retrieval and we need to consider real ground images as well. This provided us the motivation for joint feature learning.

Joint Feature Learning (third and fourth rows in Table 1 (first panel)): For joint feature learning, as explained earlier, image triads ($I_{g}$, $I_{a}$ and $I_{a^{\prime}}$) are used during training and only ($I_{g}$, $I_{a}$) pairs are used during the testing. We report an improvement of about 9% in top-1% retrieval accuracy over two-stream baseline ($I_{g}$, $I_{a}$) by joint feature learning. The improvement suggests that the synthesized aerial images contain features complementary to ground image features that facilitate the network to learn better representations for aerial images during the joint feature learning. The synthesized aerial image as an auxiliary information between the ground and aerial images is successful in forcing them to bring their feature representations closer to each other during the joint feature learning.

Since the representations for $I_{g}$, $I_{a}$ and $I_{a^{\prime}}$ were learned together during joint feature learning, we were curious to evaluate how well the feature representations for $I_{a^{\prime}}$ perform in image matching. Unsurprisingly, we obtain an improvement of about 14% in top-1% retrieval accuracy over two-stream baseline ($I_{a^{\prime}}$, $I_{a}$). This improvement further consolidated the belief that the learned features for $I_{g}$ and $I_{a^{\prime}}$ are complementary to each other and can be fused together to obtain robust descriptor for the ground image.

Feature Fusion: (fifth row in Table 1 (first panel)): The Feature Fusion approach fuses the synthesized image features with the ground image features to obtain a representative feature for the query. This provides further improvement of 3.89% in top-1% accuracy (compare fourth and fifth rows). The significance of feature fusion can be measured by about 19% improvement in top-1 retrieval accuracy over joint feature learning. This improvement further signifies that the synthesized image features are complementary to street-view image features that should be exploited to obtain better features for cross-view matching. The qualitative results are shown in Figure 3. The query ground images and the synthesized aerial images along with five closest images are shown in each row.

5.3 Comparison to Existing Methods

We compare our work with the previous approaches by [41, 46, 39, 18] on CVUSA dataset [46]. We report the top-1, top-10 and top1-% accuracies for state-of-the-art CVM-Net [18] and our methods. The results are shown in Table 1 (second panel). We observe that the Joint Feature Learning outperforms ( fourth row in Table 1 (first panel)) the previous works and is further boosted by Feature Fusion ( fifth row in Table 1 (first panel)). We achieve an overall 4.58% improvement over SOTA CVM-Net [18] for top-1% recall accuracy. We obtain significant increments of more than 26% and 18% in top-1 and top-10 accuracies over CVM-Net-I [18]. We also plot top-K recall accuracy for K = 1 to 80 for our methods as compared with previous approaches in Figure 4. It illustrates that various versions of our proposed method outperform the existing state-of-the-art approaches for all values of $K$.

5.4 Ablation Study

We conduct the following ablation studies to understand the impact of different choices made in the proposed networks. For the experiments on ablation, the joint feature learning and feature fusion networks are used with specified setups: a) single scale features - only the final layer features are matched, b) global average pooling (GAP) - GAP operation suppresses the spatial dimension of feature maps, substantially reducing the feature size, and c) weight sharing between the encoders for aerial and ground images. All these methods reduce the number of parameters used in the network.

Single Scale vs. Multi-scale Features: For this ablation, joint feature learning network with single scale features is trained first followed by experiments using the Feature Fusion network. The features after the final convolutional block (conv_8) are considered as single scale features. These are the representative features for the given input image and are used for matching. We do not employ global average pooling and weight sharing in this ablation for direct comparison of the single-scale vs. multi-scale feature representations. The scores are reported in Table 2 (first row for single scale and fourth row for multi-scale features). The results signify that features from conv_6 and conv_7 are also crucial in image matching rather than just using the features from final conv_8 layer only. The results demonstrate the importance of aggregating the multi-scale features for cross-view matching task.

**Pooling vs. No Pooling: ** We also conduct ablations on using global average pooling [23] in our experiments. Global average pooling is a popular approach to reduce the spatial dimensions of the features and consequently reduce the number of parameters in the network. We experimented with using global average pooling layer before concatenating the features from multiple scales. The result is reported in Table 2 (second row for using GAP and fourth row without using GAP, rest of the architecture being the same). We observe that the loss of spatial information in features severely impacts the retrieval performance.

Weight Sharing vs. No Weight Sharing: We believe that the two branches receiving the input images from completely different viewpoints as is the case with aerial and ground -view images should not share the weights. Even though the networks will be looking at same scene contents their representations from the two views are drastically different, thus suggesting that the networks should freely evolve their weights based on the input they receive. The results are reported in Table 2 (third row for weight sharing and fourth row for without weight sharing, remainder of the setup being the same). The numbers clearly suggest that no weight sharing is fairly an easy choice, especially looking at the difference of about 18% in Top-1 accuracies.

5.5 Cross-view Localization

We use the Orlando-Pittsburgh (OP) dataset for image based geo-localization. We want to determine the gps location of the query image by assigning it the location of closest retrieved aerial image. The query image is correctly geo-localized if it is located within a threshold distance in meters from its ground truth position.

The recall accuracy with respect to distance threshold in meters is plotted in Figure 5. We observe that our proposed Feature Fusion method can retrieve images close to its geo-location with higher accuracy than the baseline which can be attributed to its superiority in Top-1 recall.

The image retrieval examples on the OP dataset are shown in Figure 6. The ground query images are followed by the five closest aerial images. Even though the retrieved images are very similar to each other, we are able to retrieve the correct match at Top-1 position. The Top-1 recall is reported in Table 3. The results affirm that the proposed methods are generalizable to urban cities of OP dataset as well as rural areas of CVUSA dataset.

6 Conclusion

In this paper, we have presented a novel and practical approach to cross-view image retrieval by transforming the query image to target view to obtain a better scene understanding. We showed that the synthesized aerial images can be seamlessly incorporated in cross-view matching pipeline by joint feature training to bridge the domain gap between the aerial and street-view images. Also, the ground image features and the corresponding synthesized aerial image features are fused to obtain a robust descriptor of the ground image. We obtained significant improvements over state-of-the-art methods on the challenging CVUSA dataset.

Acknowledgements. The authors would like to thank Chen Chen and Yonatan Tariku for helpful discussions; Vijay Gunasekaran and Hassan Maqbool for their contribution in OP Dataset collection and the anonymous reviewers for the critical feedback.

1 Visualization and Interpretation of Features

In Figure 7, we visualize the aerial and ground image features obtained using the two-stream baseline and the proposed feature fusion methods for 100 images on the CVUSA dataset [46]. The feature representation for each image is a 1000-dimensional vector and we apply t-SNE to learn their two-dimensional embeddings for ease of visualization. The red and cyan circles close to each other or with some overlap represent the features for the ground image and its corresponding ground-truth aerial image respectively in the subplots.

The scatter-plot for features obtained using the two-stream baseline trained on ($I_{g}$, $I_{a}$) pairs is shown on the left. We observe that, for each image pair, there is less overlap between the aerial and the ground image features. We also notice that the features from different image pairs are located close to each other, with some instances of red circles overlapping each other.

The scatter-plot for the representations obtained using the proposed feature fusion method trained on image triads ($I_{g}$, $I_{a}$, $I_{a^{\prime}}$) is shown on the right subplot. We observe higher overlap between the features for ground and corresponding aerial image pairs. At the same time, we observe greater separation between the feature embeddings for different image samples.

Thus, the use of synthesized aerial images in our proposed Feature Fusion method are successful in bringing the feature representations of aerial images closer to the representations of ground images and bridging the domain gap between the images from these two drastically different views to improve the matching accuracy.

2 Failure Examples

We present some failure cases for the proposed Feature Fusion method in Figure 8. In each row, We respectively present the query ground image, corresponding synthesized aerial image, image retrieved at Top-1 position, ground-truth aerial image and a number representing the position where the ground-truth aerial image was retrieved.

Row 1 shows that ground truth aerial image consisting of water body in lower right section of the image. The ground image does not provide any information regarding water, so the image matching is challenging. The ground truth is retrieved at 13th position.

In Row 2, we can observe that the top match and ground-truth aerial images are very similar to each other. Also, the matched image has similar color distribution to query image than the ground-truth aerial image. The problem arises because the aerial and ground image pairs in the dataset are captured at different times, so have some visual differences.

Row 3 shows an example where the aerial image has houses which are not captured in street-view images due to occlusion by trees. The impact can also be observed in the corresponding synthesized image which doesn’t contain houses.

Row 4 shows that the street-view image contains a building at far distance. The building covers large region in ground-truth aerial image, which is difficult to comprehend from the street-view image. Also, this is a rare situation in the dataset with large building. So, the method fails badly, retrieving the ground-truth image at position 1700.

3 Aerial-to-Ground Image Matching

We conducted experiments for the reverse problem of Aerial-to-Ground image matching. Here, the aerial image is the query, and we attempt to find the matching ground panorama. First, we use GANs to synthesize ground level panoramas from the aerial images and then use the synthesized images in the proposed joint feature learning and feature fusion methods.

We conduct experiments for two-stream baselines, joint feature learning and feature fusion methods. The top-1, top-10 and top-1% accuracies are reported in Table 4. We obtain results similar to the numbers reported in the main paper for ground-to-aerial image matching. We also plot the top-K recall for K = 1 to 80 for the proposed method compared to the baselines in Figure 9. This affirms that our method can be applied for image matching in both directions.

The qualitative results for aerial-to-ground image matching are shown in Figure 10. The query aerial image, synthesized ground panorama followed by the three closest matches are visualized. The ground-truth panorama are shown with the green borders. We can also observe that the synthesized ground panorama are successful in transforming the semantic information from aerial to ground domain.

4 OP Dataset

The existing public datasets on cross-view image matching do not provide geo-information. Also, the images on the CVUSA dataset are collected from the rural areas that largely cover land and vegetation as shown in Figures 8 and 10. To compensate those issues, we collect a new dataset of cross-view image pairs. The images cover urban areas of Orlando and Pittsburgh. Figure 11 shows the example images of this dataset. We can observe that this dataset contains images of mainly urban areas with buildings and roads and less vegetation, contrasting to the CVUSA dataset.

We conduct experiments on the OP dataset and provide more qualitative results in Figure 11. The number below each aerial image represents its distance in meters from the query ground image. We observe that though the aerial images look very similar to each other, the proposed feature fusion method is able to retrieve the ground-truth aerial image as the closest matching image. The quantitative evaluation is provided in the main paper. We obtain similar results for top-1 accuracies on both the CVUSA and the OP dataset. This consolidates the fact that the proposed method generalizes well on both rural and urban datasets.