Geometric Image Correspondence Verification by Dense Pixel Matching

TL;DR

This paper introduces a dense pixel matching approach for verifying geometric correspondence in image retrieval, improving re-ranking accuracy by combining local and global similarity metrics with geometric consistency.

Contribution

It proposes a novel dense pixel matching and verification method that enhances image re-ranking and long-term visual localization, with architectural improvements to DGC-Net.

Findings

Outperforms state-of-the-art image retrieval methods

Demonstrates strong generalization across datasets

Improves long-term visual localization accuracy

Abstract

This paper addresses the problem of determining dense pixel correspondences between two images and its application to geometric correspondence verification in image retrieval. The main contribution is a geometric correspondence verification approach for re-ranking a shortlist of retrieved database images based on their dense pair-wise matching with the query image at a pixel level. We determine a set of cyclically consistent dense pixel matches between the pair of images and evaluate local similarity of matched pixels using neural network based image descriptors. Final re-ranking is based on a novel similarity function, which fuses the local similarity metric with a global similarity metric and a geometric consistency measure computed for the matched pixels. For dense matching our approach utilizes a modified version of a recently proposed dense geometric correspondence network (DGC-Net), which we also improve by optimizing the architecture. The proposed model and similarity metric compare favourably to the state-of-the-art image retrieval methods. In addition, we apply our method to the problem of long-term visual localization demonstrating promising results and generalization across datasets.

Figures

Tables

Table 1. Table 1 : AEPE metric for different viewpoint IDs of the HPatches dataset (lower is better).

Method	Viewpoint ID
	I	II	III	IV	V
FlowNet2 [15]	5.99	15.55	17.09	22.13	30.68
PWC-Net [41]	4.43	11.44	15.47	20.17	28.30
Rocco [30]	9.59	18.55	21.15	27.83	35.19
DGC-Net [27]	1.55	5.53	8.98	11.66	16.70
DGC-NC-UCMD-Net	1.90	5.02	9.08	10.18	13.24
DGC-NC-UCMD-Net (avg. est.)	1.51	4.46	8.66	9.59	12.62
DGC-NC-Net	1.24	4.25	8.21	9.71	13.35

Table 2. Table 2 : Localization performance on the Tokyo24/7 dataset (higher is better). Our proposed method outperforms Inloc and SIFT based geometric verification.

Methods	Recall
	r@1	r@5	r@10
DenseVLAD [43]	67.1	74.2	76.1
NetVLAD-Pitts	61.27	73.02	78.73
NetVLAD-TokyoTM [1]	71.1	83.1	86.2
SIFT	73.33	80.0	84.4
Inloc [42]	62.54	67.62	70.48
Proposed $\left(S\right)$ Pitts	71.43	82.54	85.08
Proposed $\left(S_F\right)$ Pitts	77.14	84.44	86.67

Table 3. Table 3 : Localization performance on the Aachen and CMU-Seasons datasets (higher is better). The best performance among image retrieval based approaches is highlighted as italic .

Methods	Condition, $5m,{10}^{\circ }$
	Aachen Day-Night		CMU-Seasons
	day	night	urban	suburban	park
HF-Net [34]	94.2	76.5	97.9	92.7	80.4
D2-Net [10]	93.4	74.5	-	-	-
Active Search [36]	96.6	43.9	-	-	-
NetVLAD-Pitts	81.7	64.3	78.9	77.0	63.2
Proposed	84.7	68.4	89.1	77.1	63.3

Table 4. Table 4 : Ablation study . Localization performance on the Tokyo24/7 dataset (MobileNetv2 decoder).

Methods	Recall
	r@1	r@5	r@10
Proposed $\left(S\right)$ Pitts	71.43	82.54	85.08
Proposed $\left(S\right)$ (MNetv2 enc.) Pitts	73.02	81.9	85.4
Proposed $\left(S_F\right)$ Pitts	77.14	84.44	86.67
Proposed $\left(S_F\right)$ (MNetv2 enc.) Pitts	76.51	83.17	84.13

Table 5. (a) The proposed similarity functions S 𝑆 S and S F subscript 𝑆 𝐹 S_{F} perform better strong baseline methods. An additional result after publication shows that combining our approach on SuperPoint keypoint locations gives improved retrieval results.

Methods	Recall
	r@1	r@5	r@10
Inloc [42]	62.54	67.62	70.48
NetVLAD-Pitts [1]	61.27	73.02	78.73
DenseVLAD [43]	67.10	74.20	76.10
$S$-FCNN	67.94	81.90	85.08
$S_F$-FCNN	63.49	81.59	85.71
Proposed $\left(S\right)$ Pitts	71.43	82.54	85.08
Proposed $\left(S_F\right)$ Pitts	77.14	84.44	86.67
$S_F$ + SuperPoint	83.17	86.35	87.94

Table 6. (a) The proposed similarity functions S 𝑆 S and S F subscript 𝑆 𝐹 S_{F} perform better strong baseline methods. An additional result after publication shows that combining our approach on SuperPoint keypoint locations gives improved retrieval results.

Methods	Recall
	r@1	r@5	r@10
Inloc [42]	62.54	67.62	70.48
NetVLAD-Pitts [1]	61.27	73.02	78.73
DenseVLAD [43]	67.10	74.20	76.10
$S$-FCNN	67.94	81.90	85.08
$S_F$-FCNN	63.49	81.59	85.71
Proposed $\left(S\right)$ Pitts	71.43	82.54	85.08
Proposed $\left(S_F\right)$ Pitts	77.14	84.44	86.67
$S_F$ + SuperPoint	83.17	86.35	87.94

Table 7. (b) Localization performance on the Tokyo247 dataset (higher is better).

Methods	Recall
	r@1	r@5	r@10
NetVLAD-Pitts [1]	61.27	73.02	78.73
$I$ (inliers)	56.83	78.41	83.81
$C$ (cyclically consistent inliers)	70.16	82.86	85.71
$C/I$	64.76	82.54	85.71
Proposed $\left(S\right)$ Pitts	71.43	82.54	85.08

Table 8. (c) Localization performance on the Tokyo247 dataset (higher is better).

Methods	Recall
	r@1	r@5	r@10
NetVLAD-Pitts [1]	61.27	73.02	78.73
${\log }_{10}\left(S_L\ast S\right)$	73.65	83.49	86.67
$G$	69.84	80.95	85.08
Proposed $\left(S_F\right)$ Pitts	77.14	84.44	86.67

Table 9. (d) Localization performance (Recall@1) on the Pittsburgh test dataset (higher is better). We analyze the performance of different Q 𝑄 Q and R 𝑅 R of the original similarity function ( 5 ). The baseline, NetVLAD achieves 81.59 % percent \% Recall@1

	$5/G$	${10}^{-G}$
$S_L$	77.78	81.34
$S_L\ast S$	82.04	83.70
${\log }_{10}\left(S_L\ast S\right)$	85.37	85.94

Table 10. Table 6 : Number of learnable parameters of two proposed architectures and strong baseline methods.

Model	Number of learnable parameters
PWC-Net [41]	8 749 280
GM [30]	3 271 576
DGC-Net [27]	2 675 338
Proposed (DGC-NC-Net)	2 685 079
Proposed (DGC-NC-UCMD-Net)	940 561

Equations

$$S=\frac{|C|}{|I|}\cdot exp\left(-\frac{\beta }{|C|}\right),$$

$$S_L=\sum _a\left(f_A^a\cdot f_B^a\right)m^a$$

$$S_F={\log }_{10}\left(S_L\cdot S\right)\cdot 10^{-G}$$

$$S=\frac{C}{I}\cdot exp\left(-\frac{W\cdot H}{C}\right),$$

$$S_F=\stackrel{R}{\overbrace{{\log }_{10}\left(S_L\cdot S\right)}}\cdot \underset{Q}{\underbrace{10^{-G}}}$$

Full text

1 Introduction

Image retrieval is a well studied problem in the field of computer vision and robotics with applications in place recognition [5, 13, 37], localization [21, 37, 42], autonomous driving [25], and virtual reality [28] among many others. Given a query image, the image retrieval pipeline returns a ranked list of database images according to its measure of relevance to the query image. As raw pixels are not a good representation, extensive research has gone into finding discriminative and efficient image representations. The seminal work of Sivic and Zisserman [40] proposed Bag-of-Words based image representation using SIFT [22]. Later, more advanced and efficient representations were proposed in the form of VLAD [18] descriptors and Fisher vectors [32]. More recently, off-the-shelf [3, 19, 20] and fine-tuned [1, 12, 29] convolutional neural network (CNN) representations have demonstrated great success in image retrieval. The models encode an input image to a global vector representation which leads to efficient retrieval allowing to use just a dot product as a similarity measure to obtain relevant database images. Once fine-tuned on auxiliary datasets with similar distribution as the target one, those methods have achieved state-of-the-art image retrieval performance [1, 12, 29]. However, the main limitation of such fine-tuned CNN representations are their generalization capabilities which is crucial in the context of city-scale localization where the database images can be quite similar in structure and appearance. Moreover, variations in illumination (e.g. night time queries) or occlusion can significantly affect the encoded global representations degrading retrieval performance due to lack of spatial information.

In this paper we leverage the advances of spatial geometry to obtain better ranking of the database images. To this end, we revisit the geometric verification problem in the context of image retrieval. That is, given an initial ranked list, $L$ of database images returned by a CNN model (e.g. NetVLAD), we seek to re-rank a shortlist $L^{\prime}\in L$ of images by using dense pixel correspondences [27] which are verified by the proposed similarity functions. Previously, DGC-Net [27] has been successfully applied only to positive image pairs i.e. pairs with overlapping field of view. In this work we extend its applicability to verify positive and negative image pairs in the framework of geometric verification. That is, we demonstrate how dense pixel correspondence methods such as DGC-Net can be used to improve image retrieval by geometric verification.

In summary, the contributions of this work are threefold. First, we improve the baseline DGC-Net by constraining the matching layer to be locally and globally consistent. Second, we replace multiple decoders of the original DGC-Net architecture by the proposed universal decoder, which can be shared for feature maps in different layers of the feature pyramid of DGC-Net. Third, we formulate two similarity functions, which first rank the shortlisted database images based on structural similarity and then re-rank them using appearance based similarity.

2 Related work

This work is closely related to image retrieval and image matching tasks. We provide a brief overview of existing approaches below.

Image retrieval methods can be broadly categorized into two categories: local descriptors [7, 16, 17, 23, 40] and global representations [1, 12, 29]. The approaches of the first category are based on either hand-engineered features such as SIFT [22] or learnt CNNs descriptors on the task of local image patch matching [26, 44]. Similarly, global representations methods can be further categorized into traditional hand-designed descriptors such as VLAD [18], Fisher Vectors [32], Bag-of-Words [40] and CNN based methods [1, 3, 12, 29]. Babenko et al. [3] demonstrate that the performance of off-the-shelf CNN models pre-trained on ImageNet [8] fall behind traditional local descriptors. However, when trained on an auxiliary dataset, the performance improves over such hand-engineered descriptors [1, 12, 29].

In addition to the standard retrieval approaches, there are several methods that attempt to explain the similarity between the query and top ranked database images using a geometric model [6, 24, 42]. The geometric model is estimated by fitting a simple transformation model (e.g. planar homography) to the correspondence set obtained using local descriptors such as SIFT, or off-the-shelf CNN descriptors [42]. In this work, we also use pre-trained CNN descriptors. However, in contrast to [42] which uses exhaustive nearest-neighbor search in descriptor space, we model the similarity using a learnt convolutional decoder. Moreover, [42] only uses coarse correspondence estimate, while our similarity decoder allows fine high resolution pixel level correspondence estimation. This is particularly important in city scale localization due to subtle differences in an overall similar architectural style observed in this scenario (c.f. Fig. 7).

Image matching. This task relates to the optical flow estimation problem. Recently proposed optical flow methods [15, 41] utilize a local correlation layer that performs spatially constrained matching in a coarse-to-fine manner. DGC-Net [27] extends this process of learning iterative refinement of pixel correspondences using a global correlation layer to handle wide viewpoint changes in the task of instance matching. Such a global correlation layer for instance matching has been used to estimate geometric transformations [30]. Melekhov et al. [27] demonstrate that such a method falls behind dense correspondence approaches due to the constrained range of transformations estimated by [30]. Recently, Rocco et al. [31] propose locally and globally constrained matching network on top of the global correlation layer which leads to improvement in instance and semantic matching. However, such a global correlation layer can only provide coarse correspondence estimates.

3 Method overview

Our contributions are related to the two last stages of the following three-stage image retrieval pipeline: 1) Given a query image, we retrieve a shortlist of relevant database images using a fast and scalable retrieval method based on representing images with a descriptor vector; 2) We perform dense pixel matching between the query and each shortlisted database image in a pairwise manner using a correspondence estimation network; 3) We determine a set of cyclically consistent dense pixel matches for each image pair and use them to compute a similarity metric, which provides the final re-ranking of the shortlist.

The particular architecture of the aforementioned retrieval pipeline used in this work is illustrated in Fig. 2. That is, we use NetVLAD [1] for the first stage, our own modified version of DGC-Net [27] for the second stage, and the proposed approach with a novel similarity metric for the third stage. Here NetVLAD is used for retrieval, but also other global image level descriptors could be used instead.

Our contributions related to stages 2) and 3) above are described in the following sections. The geometric verification method is presented in Section 4 and our modifications to the DGC-Net architecture are described in Section 5.

4 Geometric verification

Dense pixel correspondences produced by [27] do not take into account the underlying model explaining the 3D structure of the scene by the image pair. RANSAC [11] has been a popular method of choice to find the set of inliers from the whole correspondence set. However, dense pixel correspondences predicted by CNNs [27] are locally smooth due to the shared convolutional filters at different layers. As a result, RANSAC usually finds a large set of inliers even for non-matching image pairs. We propose two methods to eliminate these limitations in the following sections. That is, given an initial ranked shortlist $L$ of database images based on global representation similarity with the query image, we re-rank a new shortlist $L^{\prime}\subseteq{L}$ through a series of geometric verification steps (Sec. 4.1 and 4.2).

4.1 Cyclically consistent geometry

We propose a similarity cost function, $S$ that combines RANSAC based geometric model estimation with cyclic consistency. Given a dense pixel correspondence map, $O\in\mathbb{R}^{H\times W\times 2}$, RANSAC outputs a set of inliers, $I\subseteq O$ w.r.t. to a transformation model (e.g. planar homography). We then estimate the subset of inliers that are cyclically consistent, $C\subseteq I$ using forward and backward correspondence maps predicted by our network (i.e. $O^{A}$ and $O^{B}$). The cyclically consistent matches are those matches for which the combined mapping $O^{A}\!\circ\!O^{B}$ is close to an identity mapping. For geometrically dissimilar images, cyclic consistency constraint further constrains the number of inliers as the assumption here is that transformation model obtained by RANSAC may be inconsistent in forward and backward directions. We define this similarity function as follows

[TABLE]

where $\beta$ is a constant. As $|C|/|I|$ is a ratio, the exponential term is added to down-weight the similarity cost for image pairs which have less cyclically consistent correspondences in the inlier set. As $\beta$ must be greater than $|C|$, we set it to 240x240 which is the maximum value of $|C|$ as our dense correspondence network (Sec. 5) operates on fixed size images of resolution 240x240. The similarity is computed in both directions, $S^{A},S^{B}$ and the final similarity is the maximum of the two values, $S=max(S^{A},S^{B})$. The shortlist $L$ is re-ranked using $S$ resulting in the new shortlist $\hat{L}$.

4.2 Global and local similarity

Using the geometry based similarity function $S$ to re-rank the shortlist typically improves retrieval accuracy, but the retrieved list may still contain outliers as the global and local appearance similarity is not directly taken into account while computing $S$. Hence, the top-ranked database images in the geometrically verified shortlist $\hat{L}$ are passed through a second similarity function based on global and local descriptor similarity. The second similarity function is detailed below and more costly to evaluate, as it requires dense image feature extraction on high resolution images (e.g. 640x480 or higher) to obtain high resolution feature maps. On the other hand the dense correspondence estimation in Eq. 1 is performed on lower resolution images (240x240) and hence is significantly faster to compute. Therefore we have a two-stage re-ranking, where the second re-ranking is done only for a subset of top-ranked images from the first stage.

To obtain global dissimilarity $G$ we use normalized global descriptors from a pre-trained network NetVLAD [1]. The network was originally trained to learn powerful representations for image retrieval. The Euclidean distance between the global representations is defined as the global dissimilarity value $G$. To compute local similarity, we extract hypercolumn [14] features from different NetVLAD layers (see Supplementary), L2 normalize and concatenate them along channel dimension. The final features are again L2 normalized resulting in feature maps, $\tilde{F}_{A},\tilde{F}_{B}$, where $\tilde{F}\in\mathbb{R}^{H\times W\times Z}$, and $(H,W),Z$ are the image resolution and the final descriptor length. The local descriptor similarity $S_{L}$ is then obtained as:

[TABLE]

where $\cdot$ denotes inner product, $f_{A}^{a}\in{}^{w}\tilde{F}_{A}$ and $f_{B}^{a}\in\tilde{F}_{B}$ are the hypercolumn NetVLAD features at location $a$ in the warped source $\tilde{F}_{A}$ and target feature map $\tilde{F}_{B}$, and $m^{a}\in M$, where $M$ is the mask containing 1s at cyclically consistent pixels. Thus, Eq. 2 computes the cosine similarity between normalized warped source and target hypercolumn descriptors at cyclically consistent pixel locations.

The final similarity function between an image pair is a function of global dissimilarity and local similarities, $G$ and $S_{L}$ :

[TABLE]

Here, local similarity score $S_{L}$ is weighted by the similarity score $S$. We use $S_{F}$ to re-rank the top-ranked images in $\hat{L}$ to get the final shortlist $L^{\prime}$ for a given query. The $log$ term is added as a normalization to balance the local and global scores. Although there are many possible ways to combine the local and global scores, we perform an extensive evaluation (see Supplementary) and show that the current form of these equations (1 and 3) achieves the best performance.

5 Pixel correspondence estimation

To obtain dense matching between two images we use a CNN network based on the architecture of DGC-net proposed by [27]. In this section, we provide two modifications to DGC-Net leading to more compact but effective model.

5.1 Unified correspondence map decoder

In general, DGC-Net consists of a series of convolutional layers and activation functions as an encoder $E$ with $M$ layers. An input image pair $I_{A},I_{B}\in\mathbb{R}^{H\times W\times 3}$ is fed into the encoder independently to obtain a multi-resolution feature pyramid, $P=\{(F_{A}^{l},F_{B}^{l})|l=1,2,...M\}$. Here $F^{l}\in\mathbb{R}^{H_{l}\times W_{l}\times N_{l}}$ is the feature map at the output of layer $l$ of the encoder. The encoded feature maps at the top level of $P$, $(F_{A}^{M},F_{B}^{M})$ are passed through a global correlation layer, that computes exhaustive pairwise features cosine similarity. The output of correlation layer is then passed through a decoder $D_{1}$ that estimates the initial correspondence map $O_{1}$ at the same resolution as $F^{M}$. $O_{1}$ is then iteratively refined by a series of decoders $D=\{D_{2},D_{3},...D_{M}\}$ to obtain the final correspondence grid $O_{M}$ at the same resolution as input images. Each decoder, $D_{j}\in D$ takes in as input $X_{j}=\{O_{j-1},{}^{w}F_{A}^{M-j+1},F_{B}^{M-j+1}\}$, where $O_{j-1}$ is the upsampled correspondence map estimated by the previous decoder $D_{j-1}$, ${}^{w}F_{A}^{M-j+1}$ and $F_{B}^{M-j+1}$ are the warped source and target feature maps at $l=M-j+1$. However, since feature maps at level $l$ of $P$ have various number of channels, each decoder $D_{l}$ has different structure which leads to increased memory costs.

In this work, we propose a unified correspondence map decoder $D_{c}$ (UCMD) illustrated in Fig. 4. The unified decoder behaves like a recursive refinement function that operates on feature maps across different layers $l$ of $P$. More specifically, we divide the concatenated input feature maps in $X_{j}$ into $k_{j}$ non-overlapping components as shown in Fig. 4. We then propagate each of the $k_{j}$ concatenated components $X_{j}^{t},t=1,..k_{j}$ through the first convolutional layer (CB0) of our decoder, $D_{c}$. The resulting $k_{j}$ feature maps at the output of CB0 are subsequently averaged and passed through the remaining layers to obtain refined correspondence estimates $O_{j}$.

The number of inputs of CB0 is $N^{\prime}=2Q+2$, where $Q$ specifies the number of channels in feature maps ${}^{w}F_{A},F_{B}$ which are concatenated along the channel dimension. The additional 2 channels comprise of the upsampled coarser pixel correspondence map estimate from the previous layer of $P$. Therefore, $k_{l}$ is given by $\lfloor N_{l}/Q\rfloor$ where $N_{l}$ is the dimensionality of the feature maps at the current layer $l$.

Inference. During the testing phase, apart from evaluating the trained network directly we additionally follow a second strategy. We infer the pixel correspondences by feed-forwarding each $X_{j}^{t}$ through the complete decoder $D_{c}$ resulting in $k$ correspondence map estimates $O^{k}$. The process is applied to each level of the feature pyramid $P$. The mean $\mathbb{E}(O^{k})$ is used as the final pixel correspondence map estimate. This formulation was not used during training as it did not lead to convergence.

5.2 Match consistency

The global correlation layer only measures the similarities in one direction i.e. from target to source image. However, many related works in the optical flow have shown that cyclic consistency allows the network to achieve better performance. In [31], a similar kind of global correlation layer was applied with cyclic consistency and neighborhood consensus to learn optimal feature correspondence. The idea is that matches should be consistent both locally and cyclically. That is nearby matches should be locally consistent and also the matches should be consistent in both forward and backward direction. Thereby, we integrated the Neighborhood Consensus Network (NCNet) [31] in our network. In contrast to original DGC-Net, the output of the correlation layer is now passed through NCNet with learnable parameters before being feed-forwarded through the decoders $D_{M}$ and $D_{c}$ to obtain dense pixel correspondences $O$. We refer to this network as DGC-NC-UCMD-Net.

6 Experiments

We discuss the experimental settings and evaluate the proposed method on two closely related tasks, i.e. establishing dense pixel correspondences between images (image matching) and retrieval-based localization.

6.1 Image matching

For this task we compare our approach with DGC-Net [27], which can handle strong geometric transformations between two views. We use training and validation splits proposed by [27] to compare both approaches fairly. More specifically, diverse synthetic transformations (affine, TPS, and homography) have been applied to Tokyo Time Machine dataset [1] to generate  20k training samples. Similarly to [27], the proposed network has been trained by minimizing $L1$ distance between the ground-truth and estimated correspondence map $O_{i}$ at each level of the feature pyramid $P$ (c.f. Fig. 3). Details of the training procedure are given in supplementary.

We evaluate our method on HPatches dataset [4] and report the average endpoint error (AEPE) of the predicted pixel correspondence map. HPatches dataset consists of several sequences of real images with varying photometric changes. Each image sequence represents a reference image and 5 corresponding source images taken under a different viewpoint with the estimated ground-truth homography $\mathbf{H}$. As predicting a dense pixel correspondence map is closely related to optical flow estimation, we provide AEPE for strong optical flow (OF) baseline methods, i.e. FlowNet2 [15] and PWC-Net [41] respectively.

We calculate AEPE over all image sequences belonging to the same Viewpoint ID of the HPatches dataset and report the results in Tab. 1. Here, DGC-NC-Net refers to the original DGC-Net architecture complemented by NC layer (Sec. 5.2) with a set of independent decoders at each level of the spatial feature pyramid $P$. Compared to DGC-Net, this model can achieve better performance reducing the overall EPE by 20% for the most extreme viewpoint difference between the reference and source images (Viewpoint V). According to Tab. 1, DGC-NC-UCMD-Net with one universal correspondence map decoder (Sec. 5.1) falls slightly behind of DGC-NC-Net (by 12% in average across all Viewpoint IDs) but it demonstrates advantages in terms of computation and memory costs (c.f. Sec. 6.3 and Supplementary). However, DGC-NC-UCMD-Net performance can be improved further if, at inference time, rather than averaging $k$ feature maps produced by the first convolutional block of UCMD (c.f. Fig. 4) we average predicted pixel correspondence estimates for each input $k$ feature map. We refer this model as DGC-NC-UCMD-Net (avg. est.).

In addition, we report a number of correctly matched pixels between two images by calculating PCK (Percentage of Correct Keypoints) metric with different thresholds. As shown in Fig. 5, the proposed DGC-NC-* models outperform DGC-Net by about 4% and correctly match around 62% pixels for the case where geometric transformations are the most challenging (Viewpoint V).

6.2 Localization

We study the performance of our pipeline in the context of image retrieval for image based city-scale localization. For evaluating the performance of our pipeline, we consider three localization datasets: Tokyo24/7 [43], Aachen Day-Night [35], and extended CMU-Seasons [35]. For all the datasets, we follow the same procedure outlined below. For a given query we first obtain a ranked list of database images, ${L}$ based on Euclidean distance between their global NetVLAD representations, $G$. The top 100 ranked database images, $\hat{L}\subseteq{L}$ are re-ranked according to their geometric similarity score based on $S$. From these geometrically verified re-ranked database images, we pass the top 20, $L^{\prime}\subseteq\hat{L}$ through the more expensive and stricter representation similarity function, $S_{F}$. Based on this final similarity, the final re-ranking is done on $L^{\prime}$.

Localization metrics. The performance on the Tokyo24/7 dataset is evaluated using Recall@N, which is the number of queries that are correctly localized given $N$ nearest-neighbor database images returned by the model. The query is considered correctly localized if at least one of the relevant database images is presented in the top $N$ ranked database images. In contrast, the localization performance on Aachen Day-Night and extended CMU-Seasons is measured in terms of accuracy of the estimated query pose. The accuracy is defined as the percentage of queries with their estimated 6DOF pose lying within a pre-defined threshold to the ground-truth pose.

Tokyo 24/7. We compare the proposed approach with several strong baseline methods for place recognition. The hand-crafted methods are represented by DenseVLAD [43] which aggregates densely extracted SIFT descriptors [22]. As our re-ranking method aims to improve the initial ranking by NetVLAD representations, we consider NetVLAD as a baseline. In particular, we use a publicly available PyTorch implementation of NetVLAD trained on Pittsburgh dataset (NetVLAD-Pitts). As a reference, NetVLAD-Pitts obtains $85.2/94.8/97.0\%$ compared to $84.1/94.6/95.5\%$ by NetVLAD [1] on Pitts-30k [1] validation set. In addition, we also consider Inloc [42] which uses dense NetVLAD descriptors in a geometric verification setting to obtain the final shortlist of ranked database images.

The Recall@N for the baseline methods are presented in Fig. 6. Our geometric verification based pipeline achieves the state-of-the-art performance at Recall@1-10. The proposed approach significantly outperforms NetVLAD-Pitts and other baseline methods for all Recall@N thresholds (c.f. Tab. 2). Moreover, it is noteworthy that our method pushes the generalization performance of NetVLAD-Pitts above the NetVLAD-TokyoTM which was trained on images with similar distribution as Tokyo24/7. We also compared against traditional SIFT [2] based geometric verification which achieved 73.33% for Recall@1. We used COLMAP [38, 39] to extract SIFT features, followed by fundamental matrix based geometric verification to compute the inlier count.

Aachen Day-Night and Extended CMU-Seasons. Most localization systems involve an image retrieval stage where our proposed method can be directly applied. We did experiments on Aachen (day/night) and CMU Seasons datasets to show that our method retrieves more relevant database images compared to NetVLAD that leads to accurate query camera pose estimation. For each query, 20 images from the final shortlist produced by our method and NetVLAD were fed into a baseline localization pipeline, which uses a RANSAC PnP solver to register the query using 2D-3D matches (produced by performing 2D matching between the query and database images using our network and then utilizing known semi-dense point cloud for database images) and does hypothesis selection based on inlier count. We report the proportion of correctly localized queries for the threshold ($5m,10^{\circ}$) in the following. Aachen day: 81.7% (NetVLAD-Pitts), 84.7% (ours). Aachen night: 64.3% (NetVLAD-Pitts), 68.4% (ours). CMU: 78.9% (NetVLAD-Pitts), 89.1% (ours). Better accuracy in query camera pose estimation given the same localization pipeline and image matching method shows that our approach retrieves higher quality database images compared to NetVLAD-Pitts (c.f. Tab. 3). Our verification framework is generic and can be plugged in to other localization systems, such as HF-Net [34] and D2-Net [10].

Qualitative image retrieval results on Tokyo 24/7 and Aachen Day-Night datasets are illustrated in Fig. 7.

6.3 Ablation study

As the proposed UCMD decoder, $D_{c}$ is defined as a refinement function operating on the space of representation similarity, it should be invariant to the representations themselves. This allows us to replace the VGG-16 encoder with a much light weight encoder MobileNetv2 (MNetv2) [33] at test time without further re-training while keeping the same decoder trained on the features produced by VGG-16. This leads to a highly compact model. In practice, localization problem is most relevant in the context of mobile devices, thus the model compactness is crucial also at test time. This led to comparable performance on the challenging Tokyo24/7 dataset (c.f. Tab. 4) and reduced the total number of network parameters from 8M (VGG16:$\sim$7M, UCMD:$\sim$0.9M) to 1M (MNetv2:$\sim$0.07M, UCMD:$\sim$0.9M). In the latter case UCMD provides notable memory savings compared to the original DGC-net ( 10M). One feed-forward pass through the DGC-NC-UCMD-Net with MNetv2 encoder requires 60ms compared to 80ms with VGG16 encoder providing savings in computation time.

7 Conclusion

We have presented novel methods for CNN based dense pixel to pixel correspondence learning and its application to geometric verification for image retrieval. In particular, we have proposed a compact but effective CNN model for dense pixel correspondence estimation using the universal correspondence map decoder block. Due to the universal nature of the decoder, we are able to obtain memory and computational savings at evaluation time.

In addition, we have integrated the matching layer in our model with neighborhood consensus [31] which further enhances the matching performance. This modified dense correspondence model along with the proposed geometric similarity functions are then applied to improve the initial ranking of database images given by NetVLAD descriptor. We have evaluated our approach on three challenging city-scale localization datasets achieving state-of-the-art retrieval results.

Appendix

In this appendices we show additional qualitative and quantitative results of the proposed approach. In Sec. B we provide an ablation study and analyze the influence of different design choices of our method to the localization performance. We demonstrate the benefits of the unified correspondence map decoder (UCMD) compared to the architecture with multiple decoders in Sec. C Finally, qualitative localization and pixel correspondence estimation results are shown in Sec. D.

Appendix A Additional Baselines

In this work, we propose two similarity functions for geometric verification, i.e.:

[TABLE]

[TABLE]

where $I$ and $C$ is the number of inliers and cyclically consistent inliers between two $\left(W,H\right)$ images $\left(A,B\right)$, respectively; $S_{L}=\sum_{a}\left(f_{A}^{a}\cdot f_{B}^{a}\right)m^{a}$ is the local similarity between each hypercolumn ($f_{A}^{a}$ and $f_{B}^{a}$) of the NetVLAD [1] image descriptor at location $a$; $G$ is the global similarity value.

We compare our method with two baselines: i) recently proposed geometric verification pipeline Inloc [42], and ii) a neural network based method that learns the scoring functions, $S$ and $S_{F}$ given {$C,I$}, and $\{C,I,S_{L},G\}$ as input. We present more details about the baselines next.

Inloc. Inloc is a indoor localization pipeline consisting of three primary stages: i) ranking of database images by measuring global representation similarity with a given query. The global representations are obtained from the image retrieval pipeline, e.g. NetVLAD [1]; ii) a shortlist of top ranked database images are re-ranked based on geometric verification using dense CNN descriptors. The dense descriptors are obtained from different layers of the NetVLAD pipeline followed by a coarse to fine matching using nearest-neighbor search. The geometric verification is done using a standard RANSAC based inlier count. The final score is the sum of global similarity and inlier count; iii) the top ranked geometrically verified database images are fed into a pose verification stage. The final stage first estimates candidate query poses w.r.t. the current shortlisted database images. The estimated pose is then verified using view synthesis, a process requiring dense database depth maps. Our proposed geometric verification pipeline is similar to Inloc components i) and ii). The pose verification stage requires depth maps which is not always available. Therefore, we evaluate Inloc pipeline until the geometric verification stage and report results in Tab. 5(b).

Learnt similarity functions. Since both Eq. 4 and Eq. 5 are hand-crafted we provide a FCNN-based model that can learn the similarity function. More specifically, we experiment with two independent models (for $S$ and $S_{F}$) which can predict whether two images similar or not based on $C$, $I$, $S_{L}$, and $G$. Both models have similar architectures $FC(N,128)-ReLU-FC(128,128)-ReLU-FC(128,1)$, where the shorthand notation is used was the following: FC is a fully connected linear layer; $N$ is the number of input units (2 $\left\{C,I\right\}$ for $S$ and 4 $\left\{C,I,S_{L},G\right\}$ for $S_{F}$, respectively). We refer to these models as $S$-FCNN and $S_{F}$-FCNN. Both models have been trained by minimizing binary cross-entropy loss function in a supervised manner.

Results. We now compare $S$ and $S_{F}$ with Inloc geometric verification pipeline on Tokyo247 dataset. Results demonstrate that our proposed function $S$ and $S_{F}$ outperform Inloc across all Recall rates as shown in Tab. 5(a). We observed that for many query-database image pairs, Inloc fails to find any inliers. This can be attributed to significant clutter, illumination change (day-night) and occlusion in this challenging dataset. The learnt similarity functions $S$-FCNN and $S_{F}$-FCNN have very promising results and perform better than NetVLAD. In particular, $S$-FCNN has comparable performance to the proposed $S$. However, $S_{F}$-FCNN could not achieve any improvement compared to $S$-FCNN. We leave further analysis for future work.

Appendix B Ablation study

In this section we perform an ablation study on the proposed equations Eq. 4 and Eq. 5 for geometric verification. For Eq. 4, we analyze the impact of each variable, $C,I$ on retrieval performance on Tokyo247 dataset independently. The results are presented in Tab. 5(b).

Results. First we provide the ablation study for Eq. 1. Results demonstrate that simple Inlier count performs worse than the baseline NetVLAD and our proposed $S$ at Recall@1. However, the retrieval performance improves over NetVLAD for Recall@5 and Recall@10. Cyclically consistent inliers $C$ outperform NetVLAD across various Recall rates. Similarly, the ratio $C/I$ performs marginally better but it falls slightly behind of $C$ for Recall@1 (by about 6 $\%$). Both $C$ and $C/I$ perform on par with the proposed $S$ across Recall@5 and Recall@10. However, $S$ has a clear performance advantage over $C$ and $C/I$ for Recall@1 as shown in Tab. 5(b).

Now, we perform an ablation study for Eq. 5. As mentioned in the main manuscript, the proposed $S_{F}$ is used to re-rank the top 20 database images in the shortlist, $L$ as ranked by $S$. Here, we perform the final re-ranking using just the local descriptor similarity component, $\log_{10}(S_{L}*S)$, and global representation distance, $G$. Results in Tab. 5(c) demonstrate that re-ranking with $G$ decreases retrieval performance compared to the initial ranking by $S$. On the other hand, local descriptor similarity $S_{L}$ weighted by $S$ significantly improves over the baselines and initial ranking by $S$. However, the proposed combination of local and global representation similarity outperforms each individual component across all Recall rates.

The key idea here is to combine the similarity functions, $S_{L}$, $S$ and $G$. It is important to note that $S_{L}$ and $S$ are similarity functions, while $G$ is a distance function, hence, it is inversely proportional to global similarity. The inversely proportional functions, $R(S_{L},S)$ and $Q(G)$ can be combined in many different ways. We present a few in Tab. 5(d). The co-efficient (5 and 10) associated with $G$ in the columns of the Tab. 5(d) have been obtained using a grid search over the range $\left(1,10000\right)$ on Pittsburgh test dataset. In addition, we found $\hat{S}_{F}=R*Q$ performs clearly better than $\hat{S}_{F}$ = $R+Q$. Hence, we only present results for various $R$ and $Q$ for $\hat{S}_{F}=R*Q$ in Tab. 5(d). The precise form of the combination of these similarity functions has been obtained based on validation experiments on test set of Pittsburgh dataset. Tab. 5(d) shows that various possible combinations give better performance than NetVLAD which achieves 81.59 at Recall@1. The weighting with structural similarity $S$ leads to a significant boost in retrieval performance. Such a form of weighting provides good balance requiring image pairs to have high local ($S_{L}$) and structural similarity ($S$). Among the various combinations, the proposed Eq. 5 achieves the best performance (highlighted bold in Tab. 5(d)).

Appendix C The benefits of UCMD

As shown in the main manuscript, we propose the unified correspondence map decoder which leads to a compact but efficient architecture. In order to elaborate on the benefits of UCMD, here we report the average end point error averaged over all sequences of the HPatches [4] dataset obtained by each strong baseline method (PWC-Net [41], geometric matching GM [30], and DGC-Net [27]) and allocated GPU memory. The results are illustrated in Fig. 8. In contrast to DGC-NC-Net with 5 separate decoders, the proposed UCMD can significantly decrease memory footprint (by 30%) achieving comparable accuracy.

The amount of memory allocated by GM [30], DGC-Net [27], DGC-NC-Net, and DGC-NC-UCMD-Net is higher compared to PWC-Net since all those models have used pre-trained VGG-16 network as encoder. Therefore, in addition to memory consumption, we compute the total number of learnable parameters of each model and provide the results in Tab. 6.

Appendix D Qualitative results

Localization (image retrieval) performance. Fig. 9 reports an additional set of results obtained for the Tokyo24/7 dataset. Namely, it includes top-1 Nearest Neighbour (Recall@1 metric) obtained by NetVLAD [1] and our approach, respectively, for a given query. It clearly shows the proposed method improves retrieval results compared to NetVLAD and can cope with major changes in appearance (illumination changes in the scene) between the database and query images. Qualitative image retrieval results on Aachen Day-Night [35] are illustrated in Fig. 10(a).

Dense pixel correspondences are presented in Fig. 10. Each row shows one test pair from the Aachen Day-Night and Tokyo24/7 datasets, respectively. Ground truth matching keypoints are illustrated in different colors and have been used only for pixel correspondence evaluation. Keypoints of the same color are supposed to match each other. We manually indicated 3 keypoints in the target image for visualization purposes and the corresponding locations in the source image have been obtained by the proposed automatic dense matching approach. That is, given an input image pair (source and target images), our method predicts the correspondence map which is then used to obtain the location of keypoints. The results demonstrate that the proposed method can handle such challenging cases as different illumination (day/night) conditions, occlusions, and significant viewpoint changes producing accurate pixel correspondences.

Appendix E Limitations and future directions

We have demonstrated that the proposed method can localize queries under challenging conditions but it fails for very large viewpoint change (e.g. $180^{o}$ rotation while observing the same place) and significant scale change. In addition, it would be interesting to propose an end-to-end semi-supervised approach which can efficiently learn similarity functions.

The proposed similarity scores are computed at the cyclically consistent pixel locations which can be redundant due to the dense prediction by DGC-Net. A preliminary experiment showed that applying the proposed similarity functions at unique keypoint locations given by SuperPoint [9] results in improved retrieval performance as shown in Tab. 5(a). However, this comes at an additional cost of evaluating the keypoint detector. A possible future direction can be to train a joint keypoint detector and matching network.