"Don't Do That!": Guiding Embodied Systems through Large Language Model-based Constraint Generation

TL;DR

This paper introduces STPR, a framework that uses large language models to generate executable Python functions from natural language constraints, improving robotic navigation planning.

Contribution

It leverages LLMs' coding abilities to translate complex natural language constraints into structured code, enhancing interpretability and compliance in robotic planning.

Findings

STPR accurately captures complex mathematical constraints.

It ensures full compliance with constraints in simulated environments.

STPR works effectively with smaller, low-cost code LLMs.

Abstract

Recent advancements in large language models (LLMs) have spurred interest in robotic navigation that incorporates complex spatial, mathematical, and conditional constraints from natural language into the planning problem. Such constraints can be informal yet highly complex, making it challenging to translate into a formal description that can be passed on to a planning algorithm. In this paper, we propose STPR, a constraint generation framework that uses LLMs to translate constraints (expressed as instructions on ``what not to do'') into executable Python functions. STPR leverages the LLM's strong coding capabilities to shift the problem description from language into structured and interpretable code, thus circumventing complex reasoning and avoiding potential hallucinations. We show that these LLM-generated functions accurately describe even complex mathematical constraints, and apply them to point cloud representations with traditional search algorithms. Experiments in a simulated Gazebo environment show that STPR ensures full compliance across several constraints and scenarios, while having short runtimes. We also verify that STPR can be used with smaller code LLMs, making it applicable to a wide range of compact models with low inference cost.