Look No Deeper: Recognizing Places from Opposing Viewpoints under Varying Scene Appearance using Single-View Depth Estimation

TL;DR

This paper introduces a depth- and temporal-aware visual place recognition system that effectively recognizes places despite extreme viewpoint and appearance changes, outperforming existing methods on challenging benchmarks.

Contribution

The paper proposes a novel sequence-to-single matching approach using depth-filtered keypoints and temporal information, addressing the challenge of opposing viewpoints and appearance variations.

Findings

Outperforms state-of-the-art VPR techniques on benchmark datasets

Depth-filtered keypoints improve robustness to appearance changes

Sequence-based matching enhances recognition accuracy

Abstract

Visual place recognition (VPR) - the act of recognizing a familiar visual place - becomes difficult when there is extreme environmental appearance change or viewpoint change. Particularly challenging is the scenario where both phenomena occur simultaneously, such as when returning for the first time along a road at night that was previously traversed during the day in the opposite direction. While such problems can be solved with panoramic sensors, humans solve this problem regularly with limited field of view vision and without needing to constantly turn around. In this paper, we present a new depth- and temporal-aware visual place recognition system that solves the opposing viewpoint, extreme appearance-change visual place recognition problem. Our system performs sequence-to-single matching by extracting depth-filtered keypoints using a state-of-the-art depth estimation pipeline, constructing a keypoint sequence over multiple frames from the reference dataset, and comparing those keypoints to those in a single query image. We evaluate the system on a challenging benchmark dataset and show that it consistently outperforms state-of-the-art techniques. We also develop a range of diagnostic simulation experiments that characterize the contribution of depth-filtered keypoint sequences with respect to key domain parameters including degree of appearance change and camera motion.

Figures

Equations

$$p_s^{k^{\prime }}=\{p_s^k\mid u_{p_s^k}<d\}\forall k\in C,s\in S_{R_i}$$

$$r_{ji}^{k^{\prime }}=\underset{s}{\min }(1-\frac{{D^{\prime }}_j^{k^{\prime }}\cdot D_s^{k^{\prime }}}{\Vert {D^{\prime }}_j^{k^{\prime }}{\Vert }_2\Vert D_s^{k^{\prime }}{\Vert }_2})\forall k^{\prime }\in C^{\prime },s\in S_{R_i}$$

Full text

I INTRODUCTION

Visual Place Recognition (VPR) is a widely researched topic, with recent approaches benefiting from modern deep-learning techniques [1, 2, 3, 4, 5], especially for performance under challenging appearance variations. However, most of the existing literature dealing with such extreme appearance variations has only considered limited changes in viewpoint [6, 7]. In this paper, we address the more difficult problem of visual place recognition from opposite viewpoints under extreme appearance variations. Our system extracts depth-filtered keypoints using a state-of-the-art depth estimation pipeline  [8, 9, 10], constructing a keypoint sequence over multiple frames from the reference dataset, and compares those keypoints to those in a single query image.

The color-only VPR techniques for varying environmental conditions and viewpoints have shown great promise [11, 12, 13, 4]. However, the performance achieved using these systems might be limited due to the use of only RGB information. Additional information, especially depth, is an obvious potential source for improving performance. Moreover, depth is potentially useful for scale-aware metric pose estimation [9] though it is not the focus of this work. The use of single-view depth enables a good compromise between image space only methods [12, 11, 4] and those based on full 3D reconstruction [14, 15], where it can be challenging for the latter to deal with non-uniform appearance variations [16] and noisy depth measurements. In the same vein, our proposed system utilizes depth information within a sequence of images to create an on-demand local topo-metric representation [17, 18] whilst not being affected by noise in depth estimation as shown in Figure 1.

Our key contributions include:

• •

a novel sequence-to-single image matching technique111The source code will be made publicly available. that exploits within-the-image information accrued over time to maximize visual overlap between images taken from opposite viewpoints,

• •

a depth-based keypoint filtering technique to use close-range keypoints for dealing with perceptual aliasing under extreme appearance variations,

• •

performance characterization with respect to depth-based keypoint filtering, reference frame sequence length, degree of appearance change, camera motion and depth noise sensitivity.

The existing methods for visual place recognition that use a global image descriptor, such as those learnt using deep metric learning techniques [19, 4] encode spatial and structural features implicitly. While these methods form a strong baseline for top candidates retrieval for a query image, the explicit use of geometry can be beneficial for validating these top matching candidates [11, 20].

The problem of opposite viewpoints is often dealt with using sensor hardware solutions such as LiDARs [21] and panoramic cameras [22]. However, humans solve this problem regularly with a limited field-of-view vision and without having to constantly turn around. Under a similar constraint of limited field-of-view vision, dealing with 180 degree viewpoint change along with extreme appearance variations becomes a challenging research problem.

To address this challenge, we propose a sequence-to-single matching approach where a sequence of frames centered at the matching candidate location is considered for a single query image. Each of the frames in the reference candidate sequence only contributes keypoints which are within a certain depth range. As we are considering the place recognition problem for opposing viewpoints, it can be assumed that amount of visual overlap between any two matching places will only be limited to a certain depth range (on an average $30-40$ meters for maximum visual overlap [11]). Therefore, the sequence of these depth-filtered keypoints tends to capture visual information that significantly increases the overlap with the query image. We show that the proposed technique consistently outperforms the state-of-the-art methods and also improves performance over the vanilla scenario where a single reference image is used for matching. However, the performance trends depend on the environmental conditions of the compared traverses, where extreme appearance variations tend to benefit the most due to high perceptual aliasing.

II Literature Review

In visual place recognition, changing environments are a major challenge due to perceptual aliasing between the representations of places under severe appearance variations. A number of techniques have been proposed to improve the localization ability, including: leveraging temporal information [6, 23, 24], learning the appearance change over time [25, 26, 27], and extracting geometric and spatial information out of an image [28, 29, 30].

In SeqSLAM [6], utilizing sequences of recent images enables localization in challenging environments, by leveraging the scene similarity across recent images. Other sequence approaches include SMART [31], using network flows [32], and applying Conditional Random Field theory [33]. These approaches use a variety of image processing front-ends, ranging from Sum of Absolute Differences to more sophisticated methods like HOG [34] and Convolutional Neural Networks (CNNs) [35].

CNNs have demonstrated both appearance and viewpoint robustness when applied to visual place recognition [1]. In addition to using off-the-shelf CNNs [36], the place recognition task has also been learnt end-to-end [4]. In HybridNet, an off-the-shelf CNN is improved by training on appearance-change images [4, 27]. In HybridNet [27], pyramid pooling is used to improve the viewpoint robustness, however this pooling method is still defined by specific regions within a feature map. As an improvement, deep-learnt features have been intelligently extracted out of keypoint locations within these feature maps [37]. [38] proposes cross-convolutional pooling, by pooling features using the spatial position of activations in the subsequent layer. In [3], keypoint locations are extracted out of a late convolutional layer, whereas LoST [11] uses semantic information to represent places and obtain keypoint locations. LoST improves the localization ability across viewpoint variations as extreme as front to rear-view imagery. However, it relies mostly on visual semantics and spatial layout verification to achieve high performance; in this work, we delve deeper into the efficacy of CNN-based keypoints and descriptors for VPR under challenging scenarios by using depth- and temporal-aware system.

The use of geometric information for VPR has been shown to improve performance in recent works [12, 11]. In the same vein, we explore the use of depth information for improving VPR performance under vast variations in appearance and viewpoint. Depth estimation is best achieved using stereoscopic images [39], however recent advances have enabled depth estimation from monocular images. Geometric constraints [40] and non-parametric sampling [41] have been used to extract depth out of a monocular image. Improved performance can be gained by training a CNN to estimate pixel-by-pixel depth within an image [42]. In recent years, several deep networks have been proposed to estimate per-pixel-depth map and visual odometry. For instance,[43] is one of the earliest works in predicting depth by regressing with the ground truth for monocular images. The authors in [30] additionally use the motion parallax between image pairs, enabling robust depth estimation for novel scenes. Rather than using supervised learning to estimate depth, [44, 8] use unsupervised learning in an end-to-end framework. CNN-based depth estimation has been employed for dense monocular slam and semantic 3D reconstruction [45]. Our proposed approach exploits depth estimates to filter keypoints from a sequence of reference frames that are beyond a certain range for the expected visual overlap between front- and rear-view images. This forms a local topo-metric representation of the reference candidate match location which is a good compromise between full 3D reconstruction and single image only methods. However, methods like [45] will be complementary to our proposed approach.

III Proposed Approach

We use a hierarchical place matching pipeline [11] where top-N candidate matches are generated using cosine distance based descriptor comparison of a query image with the reference database using state-of-the-art retrieval methods [4, 11, 46, 38]. The query image is then matched with these top-N candidates using depth-filtered keypoints extracted from a sequence of reference frames centered at the candidate index. Figure 2 shows a flow diagram of our proposed approach as explained in subsequent sections.

III-A Keypoint and Descriptor Extraction

For a given pair of matching images, we extract keypoints and descriptors from the conv5 layer of the ResNet101 [47] as described in details in [11] and briefly here for sake of completion. For a conv5 tensor of dimension $W\times H\times C$, representing width, height, and the number of channels (or feature maps) of the tensor, a keypoint location is determined for each of the channels based on the maximally-activated 2D location within that channel. Using the same tensor, a descriptor at the keypoint location is extracted as a $C$ dimensional vector along the third axis. As only one keypoint is extracted per channel, we obtain a total of $C$ such keypoints and descriptors. For a given pair of images, the correspondences between the keypoints are assumed based on their channel index.

III-B UNsupervised MOnocular Depth Estimation

Given the reference database images, a per-pixel-depth map is estimated using the framework described in UnDEMoN [8]. The network estimates disparity maps which are then converted into depth maps using the camera parameters of the training dataset. For a keypoint $k$ at a pixel location $p_{i}^{k}$, depth $u_{p_{i}^{k}}$ is estimated from the disparity $u^{\prime}_{p_{i}^{k}}$ using the formula $u_{p_{i}^{k}}=\frac{fb}{u^{\prime}_{p_{i}^{k}}}$ where $f$ is the focal length and $b$ is the baseline distance between the stereo image pairs from the KITTI dataset [48] used to train the model. In this work, we directly used the KITTI trained models from [8] without any fine tuning performed on the datasets used in our experiments.

III-C Depth-Filtered Keypoints Sequence

For a query image, $Q_{j}$ and a top matching reference candidate $R_{i}$, keypoints ${p^{\prime}}_{j}^{k}$ and $p_{i}^{k}$ and descriptors ${D^{\prime}}_{j}^{k}$ and $D_{i}^{k}$ are extracted respectively. Further, a sequence of reference frames $S_{R_{i}}$ of length $l$ is considered across $R_{i}$, that is, $S_{R_{i}}=[R_{i-l/2},R_{i+l/2}]$ as shown in Figure 2. Then, a depth range threshold $d$ is used to filter out the keypoints that are far from the camera. Therefore, a set of keypoints is obtained from the reference sequence as below:

[TABLE]

where $k^{\prime}$ is a filtered keypoint index within $C^{\prime}\subseteq C$.

III-D Descriptor comparison

For each of these keypoints $p_{s}^{k^{\prime}}$ in the reference sequence, there exists a corresponding keypoint in the query image222The correspondences are based on the channel (feature map) index ($k^{\prime}$) within the conv5 tensor. We use cosine distance between the descriptors of these corresponding keypoints to obtain minimum descriptor distance from the query keypoint to the corresponding keypoint in any of the reference sequence frames:

[TABLE]

An average over all the filtered keypoints’ descriptor distances $r_{ji}^{k^{\prime}}$ gives the matching score $r_{ji}$ between the query image and the reference candidate.

The least scoring candidate is then considered as the final image match for the query.

IV Experimental Setup

IV-A Oxford Robotcar Dataset

The Oxford Robotcar Dataset [49] comprises traverses of Oxford city during different seasons, time of day and weather conditions, capturing images using cameras pointing in all four directions. We used an initial 2.5 km traverse from front- and rear-view cameras for four different environmental settings: Overcast Autumn, Night Autumn, Dawn Winter and Overcast Summer3332014-12-09-13-21-02, 2014-12-10-18-10-50, 2015-02-03-08-45-10 and 2015-05-19-14-06-38 respectively in [49]. We used the GPS data to sample image frames at a constant distance of approximately $2$ meters.

IV-B Ground Truth and Evaluation Method

We use GPS information to generate place recognition ground truth for the Oxford traverses. However, unlike the forward-forward image matching scenario, the opposite viewpoint place matching involves a visual offset of approximately $30-40$ meters where the visual overlap between the matching pair of images becomes maximum, as described in [11]. We use recall as a performance measure, that is equal to the number of true positives detected divided by the total number of positives in the dataset comparison. A match is considered a true positive if it lies within a certain radius of its ground truth GPS location and this radius is varied to generate the performance curves.

IV-C Comparative Study

We compare our proposed approach with two state-of-the-art methods: ‘NetVLAD’ [4] and ‘LoST-X’ [11]. LoST-X uses spatial layout matching of semantically-filtered keypoints; we also include the ‘LoST-X (No Layout Check)’ version in order to compare against semantic filtering based conv5 descriptor matching. For our proposed approach, we show results for two scenarios: one with the reference frame sequence length $l$ set to $1$ which means traditional single-to-single image matching, and the other with $l$ set to $12$ which is approximately $25$ meters for the Oxford dataset. The depth range threshold is set to $50$ and $10$ meters for similar and different environmental conditions respectively. The choice of parameters is primarily based on the previous study [11] that shows $30-40$ meters to be an average distance between two images from opposing viewpoints leading to the maximum visual overlap. Further, in Section V-B, we show performance characteristics of our proposed system with respect to these parameters that support choosing depth threshold depending on the degree of appearance variations. For a fair comparison, for all the methods, we use top $5$ match hypothesis generated by cosine distance comparison of NetVLAD descriptors.

V Results

V-A Performance comparison

Figure 4 shows performance comparison of our proposed approach against state-of-the-art methods for opposite viewpoint image matching under varying appearance of the environment. It can be observed that our proposed method consistently outperforms both NetVLAD and LoST-X. The performance gains are even more when compared against LoST-X without employing its spatial layout verification. This indicates that spatial layout matching is also a key step to attain higher performance and can potentially complement our proposed approach to improve performance even further. The performance curves also show consistently that using only a single reference image based matching has sub-optimal performance than the reference frame sequence based matching.

Qualitative Results

Figure 5 shows qualitative matches for opposite viewpoint image matching under varying environmental conditions. A sequence of frames centered at the candidate match location (index=$j$) are shown with their keypoints within a certain depth range. The rightmost column shows the sequential collection of these keypoints such that the horizontal axis topologically connects the sequence of images, however, for each image metric depth is used to separate the points. The topmost row shows depth masks for the reference image sequence used in the second row. It can be observed that due to imperfections in the estimated depth, filtering of keypoints within a given depth range is not always consistent, however, due to visual overlap between consecutive frames, this doesn’t pose a problem.

V-B Performance Characteristics

In this section, we show the performance characteristics of our proposed system with respect to the two system parameters: depth range threshold, $d$ and reference frame sequence length, $l$. These characterizations are discussed below with respect to the extent of appearance variations due to varying environmental conditions. Further, we show system’s characteristics with respect to different camera speed and the quality of depth estimation.

V-B1 Moderate Appearance Variations

Figure 6 shows that for opposite viewpoints under moderate or no changes in environmental conditions, more visual information in the form of longer reference sequence length and larger pool of keypoints within each of the reference frames, helps attain high performance. This shows that descriptors used from within the CNN are robust to moderate variations in appearance and are able to discriminate between a large number of false and true keypoint correspondences, therefore, able to utilize additional visual information.

V-B2 Extreme Appearance Variations

Figure 7 shows the performance characteristics for opposite viewpoints under extreme appearance variations from day to night and autumn to winter. It can be observed that, unlike the previous scenario of moderate appearance variations, performance tends to decrease as the visual information tends to increase, especially by allowing more keypoints per reference frame. This can be attributed to high perceptual aliasing caused due to extreme appearance variations. Therefore, accumulating keypoints within a short range of camera over the reference frame sequence helps attain optimal performance. Further within the optimal performance region, the reference frame sequence length tends to have limited effect on the performance, therefore, allowing the selection of a shorter sequence length.

V-B3 Camera Speed

Figure 8 shows variation of performance with respect to changing camera speed for the Day-Front and Night-Rear comparison. This is simulated by skipping frames within the reference frame sequence at: (a) $1\times$, (b) $2\times$, and (c) $4\times$. It can be observed that with a higher camera speed (or frame skip rate), the peak performance decreases, however, the effect of using a shorter depth range becomes more prominent.

V-B4 Quality of Depth Estimation

We use Synthia dataset [50] for evaluating the effect of quality of depth444The depth estimation network used in this paper [8] is trained on KITTI dataset [48]. for our proposed approach. For this purpose, we use the front- and rear-view images from the Dawn and Fall traverses of Sequence-02 that have very different environmental conditions, as also utilized in [11]. Figure 9 shows performance comparison using (a) ground truth depth and (b) estimated depth. It can be observed that the performance trends remain similar to those observed for Oxford dataset for opposite viewpoints and varying appearance. Further, with the use of ground truth depth, peak performance is slightly higher and performance variations are smooth with respect to the system parameters. The bottom row in Figure 9 also shows a comparison between the ground truth and estimated depth mask for an input image from Synthia reference traverse.

VI Discussion

The performance characterizations in Figure 6 and 7 show that the use of more visual information by including more distant points and using longer reference frame sequence has varied implications, depending on the variations in the scene appearance. While similar environmental settings tend to benefit from increasing visual information, performance drops drastically when images are matched across different environmental conditions. This drop in performance occurs because of perceptual aliasing among large number of keypoints matched under extreme appearance variations. Our proposed approach allows control of this visual information to achieve optimal performance by using close-range keypoints collected over multiple reference frames.

Most of the existing literature for VPR that deals with challenging appearance variations, generally deals with similar or moderate variations in viewpoints. In such cases, the above finding often gets discounted because the overall structure of the scene remains mostly similar that massively aids in matching, especially when images are down-sampled, for example, patch-normalization in SeqSLAM [6], HoG in [7], and flattened deep-learnt tensors in [51, 52, 46, 36]. Furthermore, as we use the deep metric learning method NetVLAD [4] as a baseline to generate top candidate matches, it shows that such methods, despite implicitly encoding the spatial and structural information, still have a large room for improvement. The above analysis helps in gaining insights about the behavior of deep-learnt convolutional descriptors under different appearance conditions when no assumptions are made about the overall structural similarity.

VII Conclusion and Future Work

Visual place recognition under opposing viewpoints and varying environmental conditions is a challenging problem and requires effective use of both scene appearance and scene geometry. In this paper, we proposed a sequence-to-single matching approach where an on-demand local topometric representation of reference image sequence was used to match with a single query image using depth-based keypoint filtering. We showed that our proposed system performs better than the state of the art on challenging benchmark datasets. Further, the performance characterizations also revealed that the amount of visual information can be controlled using depth-based filtering to reduce perceptual aliasing, thereby leading to optimal performance under extreme appearance variations. In future, we plan to extend our work to a visual SLAM pipeline for robust metric localization and mapping under extreme appearance and viewpoint variations.