Semantic Enhancement for Object SLAM with Heterogeneous Multimodal Large Language Model Agents

TL;DR

SEO-SLAM enhances object SLAM by integrating heterogeneous multimodal large language model agents, improving semantic accuracy and efficiency in cluttered indoor environments, and demonstrating benefits for downstream robotic tasks.

Contribution

The paper introduces a novel framework that combines heterogeneous MLLM agents with an asynchronous processing scheme and a multi-data association strategy for improved semantic SLAM.

Findings

Higher semantic accuracy and fewer false positives compared to baselines.

Significant reduction in inference time with asynchronous agents.

Potential improvements in downstream robotic assistance tasks.

Abstract

Object Simultaneous Localization and Mapping (SLAM) systems struggle to correctly associate semantically similar objects in close proximity, especially in cluttered indoor environments and when scenes change. We present Semantic Enhancement for Object SLAM (SEO-SLAM), a novel framework that enhances semantic mapping by integrating heterogeneous multimodal large language model (MLLM) agents. Our method enables scene adaptation while maintaining a semantically rich map. To improve computational efficiency, we propose an asynchronous processing scheme that significantly reduces the agents' inference time without compromising semantic accuracy or SLAM performance. Additionally, we introduce a multi-data association strategy using a cost matrix that combines semantic and Mahalanobis distances, formulating the problem as a Linear Assignment Problem (LAP) to alleviate perceptual aliasing. Experimental results demonstrate that SEO-SLAM consistently achieves higher semantic accuracy and reduces false positives compared to baselines, while our asynchronous MLLM agents significantly improve processing efficiency over synchronous setups. We also demonstrate that SEO-SLAM has the potential to improve downstream tasks such as robotic assistance. Our dataset is publicly available at: jungseokhong.com/SEO-SLAM.