Efficient Decentralized Visual Place Recognition From Full-Image Descriptors

TL;DR

This paper presents a simplified decentralized visual place recognition system using full-image descriptors like NetVLAD, reducing bandwidth and complexity compared to previous bag-of-words approaches, but with limitations in cross-environment generalization.

Contribution

It introduces a new decentralized place recognition method based on NetVLAD descriptors and k-means clustering, simplifying the system and improving scalability.

Findings

Reduces bandwidth by a factor of n with deterministic key assignment.

Simpler system compared to previous bag-of-words based methods.

Performance drops when deployment environment differs from training environment.

Abstract

In this paper, we discuss the adaptation of our decentralized place recognition method described in [1] to full image descriptors. As we had shown, the key to making a scalable decentralized visual place recognition lies in exploting deterministic key assignment in a distributed key-value map. Through this, it is possible to reduce bandwidth by up to a factor of n, the robot count, by casting visual place recognition to a key-value lookup problem. In [1], we exploited this for the bag-of-words method [3], [4]. Our method of casting bag-of-words, however, results in a complex decentralized system, which has inherently worse recall than its centralized counterpart. In this paper, we instead start from the recent full-image description method NetVLAD [5]. As we show, casting this to a key-value lookup problem can be achieved with k-means clustering, and results in a much simpler system than [1]. The resulting system still has some flaws, albeit of a completely different nature: it suffers when the environment seen during deployment lies in a different distribution in feature space than the environment seen during training.

Full text

I Introduction

In [1], we have shown how the data exchange incurred in decentralized visual place recognition can be reduced by a factor of up to $n$, the robot count. This can be achieved by casting the place recognition problem to a key-value lookup problem, which can be efficiently distributed using deterministic key-to-peer assignment, as is for example common in distributed hash tables [6, 7]. In [1], we have thus cast the bag-of-words (BoW) place recognition method [3, 4] used in [8, 9]. In broad strokes, this is how the resulting method works:

Before deployment, deterministically assign words of the visual vocabulary to the different robots. 2. 2.

When querying place recognition of an image frame, calculate the BoW vector and split it up into partial BoW vectors such that one partial BoW vector can be sent to each robot $r$, containing the coefficients of the words assigend to $r$. 3. 3.

The robots receive and process each their own partial query, returning the identity of the single frame which best matches the query frame according to the partial BoW vector. They also store the query, making it available as a result for subsequent queries. 4. 4.

Gather all partial results and determine which frame is most consistently returned as result. 5. 5.

Send a full query to the robot that has observed that frame for geometric verification.

We have shown that this methods results in a bandwidth reduction of up to $n$ (depending on the network infrastructure), while reducing recall by $10-20\%$ depending on the robot count. A lot of the recall reduction is due to steps 3) and 4) of the method, which are based on a simplifying assumption that we do not yet fully understand. See Sec. IV C. and Fig. 4 of [1] for a detialed discussion.

II Methodology

In this whitepaper, we instead propose to use a full-image descriptor place recognition method as a basis. In particular, we use the recent, deep-learning based NetVLAD method [5] which has been shown to perform excellently even under severe appearance and viewpoint changes. Indeed, as can be seen in the centralized evaluation of this method (Fig. 1), its recall qualitatively looks better than the one of the BoW method we used in [1].

NetVLAD uses a deep neural network to calculate a low-dimensional feature vector $\vec{v}\in\mathbb{R}^{d}$ from an input image. Place matches can then be found by looking for the nearest vectors of other images according to the $\ell_{2}$ distance.

This method can now efficiently be decentralized in the following way:

Before deployment, cluster the feature vector space and assign each cluster center to a robot. 2. 2.

When querying place recognition of an image frame, calculate the feature vector and send it as query to only the robot assigned to the corresponding cluster. 3. 3.

That robot processes the query, stores it for future reference, and replies with the best matching frame identifier. 4. 4.

Send a full query to the robot that has observed that frame for geometric verification.

Evidently, this method is leaner than the one proposed in [1]: data is sent to only one robot, and no assumptions on the fidelity of partial responses are made. In this preliminary work, we use k-means clustering cite! as clustering method for step 1). The clustering is trained on image data from the Oxford RobotCar dataset [10].

III Experiments

Since this is preliminary work, we use a simpler evaluation methodology than the one in [1]: firstly, we don’t actully implement the method on multiple processes as we did there. It is evident form the method that it needs $n$ times less data exchange than if all queries were sent to all robots. To evaluate the place recognition performance the method would have if deployed on a group of robots, we simply exclude all images that are not in the same cluster as the query from the pool of possible responses to a query.

Secondly, we don’t perform geometric verification. To keep a fair evaluation, we evaluate precision and recall for all possible feature vector distance thresholds and consider the area under that curve (AUC) as metric for place recognition performance, see Fig. 2.

We use then NetVLAD feature vector dimension $d=128$ (tunable thanks to a final layer that does principal component analysis). The method is evaluated on KITTI 00 [11] by splitting the sequence into $n$ sub-sequences, one per robot.

IV Results

Fig. 3 shows relative AUC (decentralized to centralized) of the method when applied to groups of $n\in[2,20]$ robots.

Recall suffers if the true match of a query is not in the same cluster as the query. It would seem that the performance of the decentralized NetVLAD method is only marginally better than the performance of the decentralized BoW method (see Fig. 7 in [1]). Consider however that as qualitatively seen in Fig. 1, NetVLAD already has a higher recall than BoW in the first place. Furthermore, the method uses far less bandwidth for its distributed query than the BoW method. Recall from Table II in [1] that its distributed query size is $16$ kilobytes plus overhead from sending the query to $n$ robots. This method, when using single precision, only needs $d\times 4$ bytes per query, so $512$ bytes with $d=128$, plus overhead from only sending to one robot.

However, as it turns out, this method at its current state is very bad at balancing the computational and network load among the robots. In Fig. 3 we furthermore report the worst balance ratio, a measure for how much more queries the busiest robot receives compared to how much it would receive if the queries were perfectly balanced. As we can see, the busiest robot handles up to half of all queries! This can be traced back to bad clustering.

Why is the clustering bad? This can happen when the distribution of feature vectors in the deployment dataset is not the same as the distribution of vectors in the cluster training dataset. As it turns out, the features in KITTI 00 only occupy a subspace of what the features in the RobotCar dataset occupy, see Fig. 4.

This is however not a problem of the training but a problem that is implicit in the method: The training set should be as general as possible (i.e. covering as many different environments as possible), while it is perfectly possible for the feature vectors encountered during deployment to stem only from a very specific type of environment.

In future work we will try to overcome the balancing issue, and try to see what parts of the pipeline can be improved to yield better performance.

Acknowledgement

We would like to thank Antonio Loquercio for the code reviews and helpful feedback.