Using VLM Reasoning to Constrain Task and Motion Planning

TL;DR

This paper introduces VIZ-COAST, a novel approach that uses large pretrained Vision-Language Models to identify potential refinement issues in task and motion planning beforehand, significantly improving planning efficiency and reliability.

Contribution

The paper presents VIZ-COAST, a method that leverages vision-language models to predict refinement feasibility, reducing the need for costly replanning in task and motion planning.

Findings

Drastically reduces planning times in TAMP domains

Eliminates downward refinement failures in some cases

Generalizes across diverse domain instances

Abstract

In task and motion planning, high-level task planning is done over an abstraction of the world to enable efficient search in long-horizon robotics problems. However, the feasibility of these task-level plans relies on the downward refinability of the abstraction into continuous motion. When a domain's refinability is poor, task-level plans that appear valid may ultimately fail during motion planning, requiring replanning and resulting in slower overall performance. Prior works mitigate this by encoding refinement issues as constraints to prune infeasible task plans. However, these approaches only add constraints upon refinement failure, expending significant search effort on infeasible branches. We propose VIZ-COAST, a method of leveraging the common-sense spatial reasoning of large pretrained Vision-Language Models to identify issues with downward refinement a priori, bypassing the need to fix these failures during planning. Experiments on three challenging TAMP domains show that our approach is able to extract plausible constraints from images and domain descriptions, drastically reducing planning times and, in some cases, eliminating downward refinement failures altogether, generalizing to a diverse range of instances from the broader domain.