Bridging Text and Vision: A Multi-View Text-Vision Registration Approach for Cross-Modal Place Recognition

TL;DR

This paper introduces Text4VPR, a novel multi-view text-vision registration method that uses textual descriptions for place recognition, achieving high accuracy and demonstrating the feasibility of location identification solely from language descriptions.

Contribution

It is the first approach to match textual descriptions with multi-view images for place recognition, utilizing a frozen language model and novel alignment techniques.

Findings

Achieved 57% top-1 accuracy on Street360Loc dataset.

Demonstrated the feasibility of text-based place recognition.

Established a robust baseline for future research.

Abstract

Mobile robots necessitate advanced natural language understanding capabilities to accurately identify locations and perform tasks such as package delivery. However, traditional visual place recognition (VPR) methods rely solely on single-view visual information and cannot interpret human language descriptions. To overcome this challenge, we bridge text and vision by proposing a multiview (360{\deg} views of the surroundings) text-vision registration approach called Text4VPR for place recognition task, which is the first method that exclusively utilizes textual descriptions to match a database of images. Text4VPR employs the frozen T5 language model to extract global textual embeddings. Additionally, it utilizes the Sinkhorn algorithm with temperature coefficient to assign local tokens to their respective clusters, thereby aggregating visual descriptors from images. During the training stage, Text4VPR emphasizes the alignment between individual text-image pairs for precise textual description. In the inference stage, Text4VPR uses the Cascaded Cross-Attention Cosine Alignment (CCCA) to address the internal mismatch between text and image groups. Subsequently, Text4VPR performs precisely place match based on the descriptions of text-image groups. On Street360Loc, the first text to image VPR dataset we created, Text4VPR builds a robust baseline, achieving a leading top-1 accuracy of 57% and a leading top-10 accuracy of 92% within a 5-meter radius on the test set, which indicates that localization from textual descriptions to images is not only feasible but also holds significant potential for further advancement, as shown in Figure 1.