Component Selection for Craft Assembly Tasks

TL;DR

This paper introduces the Craft Assembly Task, a robotic assembly challenge inspired by handmade crafts, focusing on selecting suitable objects from RGB images to assemble an accurate representation of a target object.

Contribution

It proposes a novel approach combining mask segmentation, template retrieval, pose optimization, shape simplification, and a search algorithm for object selection in craft assembly tasks.

Findings

Achieves comparable results to baseline methods in scene object selection

Demonstrates effectiveness in real-world scenarios

Provides a new framework for craft-inspired robotic assembly

Abstract

Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labelled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.