Language-Conditioned Path Planning

TL;DR

This paper introduces Language-Conditioned Path Planning, enabling robots to plan paths that incorporate contact with objects, using a novel collision prediction method trained on minimal data, to enhance manipulation capabilities.

Contribution

The paper presents LACO, a novel language-conditioned collision function that predicts collisions from images and language prompts, allowing contact-aware path planning without extensive annotations.

Findings

LACO effectively predicts collisions using minimal input data.

The approach enables safe, contact-rich path planning in simulation and real-world scenarios.

It allows robots to interact with objects safely by understanding when contact is acceptable.

Abstract

Contact is at the core of robotic manipulation. At times, it is desired (e.g. manipulation and grasping), and at times, it is harmful (e.g. when avoiding obstacles). However, traditional path planning algorithms focus solely on collision-free paths, limiting their applicability in contact-rich tasks. To address this limitation, we propose the domain of Language-Conditioned Path Planning, where contact-awareness is incorporated into the path planning problem. As a first step in this domain, we propose Language-Conditioned Collision Functions (LACO) a novel approach that learns a collision function using only a single-view image, language prompt, and robot configuration. LACO predicts collisions between the robot and the environment, enabling flexible, conditional path planning without the need for manual object annotations, point cloud data, or ground-truth object meshes. In both simulation and the real world, we demonstrate that LACO can facilitate complex, nuanced path plans that allow for interaction with objects that are safe to collide, rather than prohibiting any collision.