ExTraCT -- Explainable Trajectory Corrections from language inputs using Textual description of features

TL;DR

ExTraCT is a modular framework that uses large language models and textual feature descriptions to generate natural language-guided trajectory corrections for robots, improving generalization and user preference.

Contribution

This work introduces a novel modular approach combining LLMs and trajectory deformation functions for natural language-based robot trajectory corrections, enabling better generalization.

Findings

Trajectories corrected with ExTraCT are more accurate.

Users preferred ExTraCT-corrected trajectories in 80% of cases.

System demonstrates versatility in manipulation and assistive tasks.

Abstract

Natural language provides an intuitive and expressive way of conveying human intent to robots. Prior works employed end-to-end methods for learning trajectory deformations from language corrections. However, such methods do not generalize to new initial trajectories or object configurations. This work presents ExTraCT, a modular framework for trajectory corrections using natural language that combines Large Language Models (LLMs) for natural language understanding and trajectory deformation functions. Given a scene, ExTraCT generates the trajectory modification features (scene-specific and scene-independent) and their corresponding natural language textual descriptions for the objects in the scene online based on a template. We use LLMs for semantic matching of user utterances to the textual descriptions of features. Based on the feature matched, a trajectory modification function is applied to the initial trajectory, allowing generalization to unseen trajectories and object configurations. Through user studies conducted both in simulation and with a physical robot arm, we demonstrate that trajectories deformed using our method were more accurate and were preferred in about 80\% of cases, outperforming the baseline. We also showcase the versatility of our system in a manipulation task and an assistive feeding task.