LatentBKI: Open-Dictionary Continuous Mapping in Visual-Language Latent Spaces with Quantifiable Uncertainty

TL;DR

LatentBKI is a probabilistic mapping method that integrates vision-language models into a continuous, open-vocabulary semantic mapping framework with quantifiable uncertainty, suitable for complex robotic tasks.

Contribution

It introduces LatentBKI, a novel Bayesian Kernel Inference-based algorithm that enables open-dictionary semantic mapping in visual-language latent spaces with uncertainty quantification.

Findings

Outperforms similar semantic mapping methods on Matterport3D and Semantic KITTI datasets.

Maintains probabilistic benefits while supporting open-vocabulary queries.

Proven effective in real-world indoor environments.

Abstract

This paper introduces a novel probabilistic mapping algorithm, LatentBKI, which enables open-vocabulary mapping with quantifiable uncertainty. Traditionally, semantic mapping algorithms focus on a fixed set of semantic categories which limits their applicability for complex robotic tasks. Vision-Language (VL) models have recently emerged as a technique to jointly model language and visual features in a latent space, enabling semantic recognition beyond a predefined, fixed set of semantic classes. LatentBKI recurrently incorporates neural embeddings from VL models into a voxel map with quantifiable uncertainty, leveraging the spatial correlations of nearby observations through Bayesian Kernel Inference (BKI). LatentBKI is evaluated against similar explicit semantic mapping and VL mapping frameworks on the popular Matterport3D and Semantic KITTI datasets, demonstrating that LatentBKI maintains the probabilistic benefits of continuous mapping with the additional benefit of open-dictionary queries. Real-world experiments demonstrate applicability to challenging indoor environments.