Meta-Optimization and Program Search using Language Models for Task and Motion Planning

TL;DR

This paper introduces a novel meta-optimization approach using language models for task and motion planning, effectively bridging high-level symbolic reasoning and low-level control in robotics.

Contribution

It proposes a new interface employing program search and zero-order optimization to enhance foundation model-based TAMP methods.

Findings

Improved planning times over prior TAMP methods

Enhanced performance on object manipulation tasks

Effective integration of symbolic and continuous control

Abstract

Intelligent interaction with the real world requires robotic agents to jointly reason over high-level plans and low-level controls. Task and motion planning (TAMP) addresses this by combining symbolic planning and continuous trajectory generation. Recently, foundation model approaches to TAMP have presented impressive results, including fast planning times and the execution of natural language instructions. Yet, the optimal interface between high-level planning and low-level motion generation remains an open question: prior approaches are limited by either too much abstraction (e.g., chaining simplified skill primitives) or a lack thereof (e.g., direct joint angle prediction). Our method introduces a novel technique employing a form of meta-optimization to address these issues by: (i) using program search over trajectory optimization problems as an interface between a foundation model and robot control, and (ii) leveraging a zero-order method to optimize numerical parameters in the foundation model output. Results on challenging object manipulation and drawing tasks confirm that our proposed method improves over prior TAMP approaches.