Visual Sculpting: Visually-Aligned Planning Representations for Long-Horizon Robot Clay Sculpting

TL;DR

This paper introduces a visually-aligned dynamics model for robotic clay sculpting, enabling long-horizon planning that captures textures and lighting, improving over prior sparse representations.

Contribution

It presents a novel visual representation for deformable object manipulation, facilitating effective long-horizon planning in robotic clay sculpting tasks.

Findings

The dynamics model performs comparably to state-of-the-art methods.

Planning in a visual representation benefits long-horizon sculpting.

The method works across three different deformable materials.

Abstract

Clay sculpting is a nuanced, artistic task involving dexterous manipulation with long-horizon planning to achieve high-level goals. As a robotics problem, we formulate clay sculpting as a shape-to-shape matching challenge. Prior deformable object manipulation work either requires retraining a policy per goal or relies on dynamics models which represent state as sparse point clouds which do not capture important clay features, such as textures, well. We present a method for modeling the dynamics of deformable materials and planning for robotic sculpting in a representation that is visually-aligned, capturing lighting and texture features. With three different deformable materials and various end-effectors, we demonstrate that our dynamics model is comparable in performance to the state-of-the-art with the added benefit of being compatible with visual planning. Our actions are represented as parametrized pushes into clay with a single end-effector, which proved to be suitable for long-horizon (>100 actions) clay relief sculptures. Lastly, we show the benefits of planning in a visually-aligned representation, but also provide analysis providing evidence as to why this representation is challenging to plan in compared to 3D representations.