Multi-modal Visual Place Recognition in Dynamics-Invariant Perception Space

TL;DR

This paper introduces a multi-modal fusion approach combining semantic and visual data in a dynamics-invariant space to enhance visual place recognition robustness in dynamic environments using a novel deep learning architecture.

Contribution

It presents a new deep learning architecture for static semantic segmentation and image recovery, and a multi-modal encoding method for improved place recognition in dynamic settings.

Findings

Effective in dynamic environments

Robustness demonstrated through extensive experiments

Multi-modal fusion improves recognition accuracy

Abstract

Visual place recognition is one of the essential and challenging problems in the fields of robotics. In this letter, we for the first time explore the use of multi-modal fusion of semantic and visual modalities in dynamics-invariant space to improve place recognition in dynamic environments. We achieve this by first designing a novel deep learning architecture to generate the static semantic segmentation and recover the static image directly from the corresponding dynamic image. We then innovatively leverage the spatial-pyramid-matching model to encode the static semantic segmentation into feature vectors. In parallel, the static image is encoded using the popular Bag-of-words model. On the basis of the above multi-modal features, we finally measure the similarity between the query image and target landmark by the joint similarity of their semantic and visual codes. Extensive experiments demonstrate the effectiveness and robustness of the proposed approach for place recognition in dynamic environments.