LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models

TL;DR

LLM-Craft introduces a novel approach that uses large language models to reason about and generate deformation sequences for complex shape crafting, mimicking human-like reasoning in sculpting tasks.

Contribution

This work is the first to leverage large language models for complex deformable object interactions in robotic crafting, demonstrating shape-based reasoning capabilities.

Findings

LLM-Craft successfully creates simple letter shapes.

Different rollout strategies impact performance.

Explicit goal shape images influence results.

Abstract

When humans create sculptures, we are able to reason about how geometrically we need to alter the clay state to reach our target goal. We are not computing point-wise similarity metrics, or reasoning about low-level positioning of our tools, but instead determining the higher-level changes that need to be made. In this work, we propose LLM-Craft, a novel pipeline that leverages large language models (LLMs) to iteratively reason about and generate deformation-based crafting action sequences. We simplify and couple the state and action representations to further encourage shape-based reasoning. To the best of our knowledge, LLM-Craft is the first system successfully leveraging LLMs for complex deformable object interactions. Through our experiments, we demonstrate that with the LLM-Craft framework, LLMs are able to successfully create a set of simple letter shapes. We explore a variety of rollout strategies, and compare performances of LLM-Craft variants with and without an explicit goal shape images. For videos and prompting details, please visit our project website: https://sites.google.com/andrew.cmu.edu/llmcraft/home