SENT Map -- Semantically Enhanced Topological Maps with Foundation Models

TL;DR

SENT-Map introduces a semantically enhanced topological map for indoor environments, leveraging foundation models and JSON format to improve autonomous navigation and manipulation.

Contribution

It presents a novel framework combining semantic mapping with foundation models, enabling effective indoor environment planning with small FMs.

Findings

Semantic enhancement improves planning success in indoor environments.

JSON-based representation allows easy editing and understanding by humans and FMs.

Small foundation models can effectively plan using SENT-Map.

Abstract

We introduce SENT-Map, a semantically enhanced topological map for representing indoor environments, designed to support autonomous navigation and manipulation by leveraging advancements in foundational models (FMs). Through representing the environment in a JSON text format, we enable semantic information to be added and edited in a format that both humans and FMs understand, while grounding the robot to existing nodes during planning to avoid infeasible states during deployment. Our proposed framework employs a two stage approach, first mapping the environment alongside an operator with a Vision-FM, then using the SENT-Map representation alongside a natural-language query within an FM for planning. Our experimental results show that semantic-enhancement enables even small locally-deployable FMs to successfully plan over indoor environments.