OVITA: Open-Vocabulary Interpretable Trajectory Adaptations

TL;DR

OVITA is a novel framework that uses large language models to enable natural language-based, interpretable, and flexible trajectory adaptation for robots in dynamic, unstructured environments, enhancing user interaction and control.

Contribution

The paper introduces OVITA, a framework that integrates multiple pre-trained LLMs for open-vocabulary, language-driven trajectory adaptation in robots, allowing intuitive waypoint adjustments without expert knowledge.

Findings

Effective in simulation and real-world tasks

Supports diverse robotic platforms including manipulators and drones

Enables intuitive, natural language-based trajectory modifications

Abstract

Adapting trajectories to dynamic situations and user preferences is crucial for robot operation in unstructured environments with non-expert users. Natural language enables users to express these adjustments in an interactive manner. We introduce OVITA, an interpretable, open-vocabulary, language-driven framework designed for adapting robot trajectories in dynamic and novel situations based on human instructions. OVITA leverages multiple pre-trained Large Language Models (LLMs) to integrate user commands into trajectories generated by motion planners or those learned through demonstrations. OVITA employs code as an adaptation policy generated by an LLM, enabling users to adjust individual waypoints, thus providing flexible control. Another LLM, which acts as a code explainer, removes the need for expert users, enabling intuitive interactions. The efficacy and significance of the proposed OVITA framework is demonstrated through extensive simulations and real-world environments with diverse tasks involving spatiotemporal variations on heterogeneous robotic platforms such as a KUKA IIWA robot manipulator, Clearpath Jackal ground robot, and CrazyFlie drone.