Prompt2Craft: Generating Functional Craft Assemblies with LLMs

TL;DR

This paper introduces the Craft Assembly Task, a robotic assembly challenge inspired by handmade crafts, focusing on selecting and assembling objects to replicate a target shape from images using neural networks and shape simplification.

Contribution

It presents a novel task formulation, a pipeline combining segmentation, template retrieval, pose optimization, shape simplification, and correspondence search, advancing robotic craft assembly capabilities.

Findings

Achieves comparable results to baseline methods in scene assembly

Demonstrates effective shape simplification for assembly tasks

Provides qualitative results in real-world scenarios

Abstract

Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labeled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.