FM-Loc: Using Foundation Models for Improved Vision-based Localization

TL;DR

FM-Loc introduces a novel vision-based localization method leveraging foundation models like CLIP and GPT-3 to create semantic image descriptors, improving robustness to environmental changes without training.

Contribution

The paper presents a new localization approach using foundation models for semantic scene understanding, eliminating the need for training or fine-tuning.

Findings

Effective in indoor environments with viewpoint and object placement changes

No training or fine-tuning required for the approach

Demonstrates robustness in real-world scenarios

Abstract

Visual place recognition is essential for vision-based robot localization and SLAM. Despite the tremendous progress made in recent years, place recognition in changing environments remains challenging. A promising approach to cope with appearance variations is to leverage high-level semantic features like objects or place categories. In this paper, we propose FM-Loc which is a novel image-based localization approach based on Foundation Models that uses the Large Language Model GPT-3 in combination with the Visual-Language Model CLIP to construct a semantic image descriptor that is robust to severe changes in scene geometry and camera viewpoint. We deploy CLIP to detect objects in an image, GPT-3 to suggest potential room labels based on the detected objects, and CLIP again to propose the most likely location label. The object labels and the scene label constitute an image descriptor that we use to calculate a similarity score between the query and database images. We validate our approach on real-world data that exhibit significant changes in camera viewpoints and object placement between the database and query trajectories. The experimental results demonstrate that our method is applicable to a wide range of indoor scenarios without the need for training or fine-tuning.