Language-EXtended Indoor SLAM (LEXIS): A Versatile System for Real-time Visual Scene Understanding

TL;DR

LEXIS is a real-time indoor SLAM system that leverages large language models for versatile scene understanding, enabling adaptive room classification, segmentation, and improved place recognition in diverse environments.

Contribution

Introduces LEXIS, a novel SLAM system integrating open-vocabulary LLMs for flexible semantic understanding and place recognition in indoor environments.

Findings

Successfully categorizes rooms with varying layouts and dimensions.

Outperforms state-of-the-art in room classification and segmentation.

Achieves comparable performance to SOTA in place recognition and trajectory estimation.

Abstract

Versatile and adaptive semantic understanding would enable autonomous systems to comprehend and interact with their surroundings. Existing fixed-class models limit the adaptability of indoor mobile and assistive autonomous systems. In this work, we introduce LEXIS, a real-time indoor Simultaneous Localization and Mapping (SLAM) system that harnesses the open-vocabulary nature of Large Language Models (LLMs) to create a unified approach to scene understanding and place recognition. The approach first builds a topological SLAM graph of the environment (using visual-inertial odometry) and embeds Contrastive Language-Image Pretraining (CLIP) features in the graph nodes. We use this representation for flexible room classification and segmentation, serving as a basis for room-centric place recognition. This allows loop closure searches to be directed towards semantically relevant places. Our proposed system is evaluated using both public, simulated data and real-world data, covering office and home environments. It successfully categorizes rooms with varying layouts and dimensions and outperforms the state-of-the-art (SOTA). For place recognition and trajectory estimation tasks we achieve equivalent performance to the SOTA, all also utilizing the same pre-trained model. Lastly, we demonstrate the system's potential for planning.