Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions

TL;DR

This paper introduces a novel approach combining large language models and robot actions to understand and utilize the physical properties of unseen deformable objects for task planning, especially in bin-packing scenarios.

Contribution

It presents a new LLM-based method for probing physical properties of unseen deformable objects through robot interactions, enhancing task planning capabilities.

Findings

The method successfully identifies properties of deformable objects.

Properties are effectively used for task planning in bin-packing.

The approach improves handling of unseen deformable objects in robotics.

Abstract

In this paper, we consider the problem of understanding the physical properties of unseen objects through interactions between the objects and a robot. Handling unseen objects with special properties such as deformability is challenging for traditional task and motion planning approaches as they are often with the closed world assumption. Recent results in Large Language Models (LLMs) based task planning have shown the ability to reason about unseen objects. However, most studies assume rigid objects, overlooking their physical properties. We propose an LLM-based method for probing the physical properties of unseen deformable objects for the purpose of task planning. For a given set of object properties (e.g., foldability, bendability), our method uses robot actions to determine the properties by interacting with the objects. Based on the properties examined by the LLM and robot actions, the LLM generates a task plan for a specific domain such as object packing. In the experiment, we show that the proposed method can identify properties of deformable objects, which are further used for a bin-packing task where the properties take crucial roles to succeed.