Bootstrapped LLM Semantics for Context-Aware Path Planning

TL;DR

This paper introduces a framework that leverages large language models as semantic sensors to improve context-aware path planning in human-centric environments, integrating natural language prompts with classical planning methods.

Contribution

The work presents a novel approach that uses LLMs as stochastic semantic sensors, enabling robots to adapt their navigation based on semantic risk assessments derived from language prompts.

Findings

Method effectively modulates robot movement in simulated environments

Bayesian bootstrap approximates risk posterior from LLM judgments

Framework adapts to explicit prompts and implicit context

Abstract

Prompting robots with natural language (NL) has largely been studied as what task to execute (goal selection, skill sequencing) rather than how to execute that task safely and efficiently in semantically rich, human-centric spaces. We address this gap with a framework that turns a large language model (LLM) into a stochastic semantic sensor whose outputs modulate a classical planner. Given a prompt and a semantic map, we draw multiple LLM "danger" judgments and apply a Bayesian bootstrap to approximate a posterior over per-class risk. Using statistics from the posterior, we create a potential cost to formulate a path planning problem. Across simulated environments and a BIM-backed digital twin, our method adapts how the robot moves in response to explicit prompts and implicit contextual information. We present qualitative and quantitative results.