Planning and Reasoning with 3D Deformable Objects for Hierarchical Text-to-3D Robotic Shaping

TL;DR

This paper introduces a novel autonomous system for text-guided 3D sculpting of deformable objects, combining large language models and point cloud-based action prediction to create shapes from text prompts without explicit goals.

Contribution

It presents the first coarse-to-fine autonomous sculpting pipeline that operates in real-world settings, integrating sub-goal generation and deformation actions for text-to-3D shape creation.

Findings

Successfully creates simple 3D shapes from text prompts.

Demonstrates the effectiveness of a hierarchical sculpting approach.

Evaluates and compares similarity metrics for assessing sculpting success.

Abstract

Deformable object manipulation remains a key challenge in developing autonomous robotic systems that can be successfully deployed in real-world scenarios. In this work, we explore the challenges of deformable object manipulation through the task of sculpting clay into 3D shapes. We propose the first coarse-to-fine autonomous sculpting system in which the sculpting agent first selects how many and where to place discrete chunks of clay into the workspace to create a coarse shape, and then iteratively refines the shape with sequences of deformation actions. We leverage large language models for sub-goal generation, and train a point cloud region-based action model to predict robot actions from the desired point cloud sub-goals. Additionally, our method is the first autonomous sculpting system that is a real-world text-to-3D shaping pipeline without any explicit 3D goals or sub-goals provided to the system. We demonstrate our method is able to successfully create a set of simple shapes solely from text-based prompting. Furthermore, we explore rigorously how to best quantify success for the text-to-3D sculpting task, and compare existing text-image and text-point cloud similarity metrics to human evaluations for this task. For experimental videos, human evaluation details, and full prompts, please see our project website: https://sites.google.com/andrew.cmu.edu/hierarchicalsculpting