Benchmarking Off-The-Shelf Solutions to Robotic Assembly Tasks

TL;DR

This paper evaluates off-the-shelf industrial solutions on a new benchmark for robotic assembly, revealing their limitations and establishing a baseline for future research in robotic manipulation tasks.

Contribution

It provides the first objective benchmark performance of off-the-shelf solutions on NIST Assembly Task Boards, highlighting key challenges and limitations.

Findings

OTS solutions are expertise-dependent and have limited applicability.

Interoperability and scene awareness are lacking in current solutions.

The study establishes a baseline for future benchmarking efforts.

Abstract

In recent years, many learning based approaches have been studied to realize robotic manipulation and assembly tasks, often including vision and force/tactile feedback. However, it remains frequently unclear what is the baseline state-of-the-art performance and what are the bottleneck problems. In this work, we evaluate some off-the-shelf (OTS) industrial solutions on a recently introduced benchmark, the National Institute of Standards and Technology (NIST) Assembly Task Boards. A set of assembly tasks are introduced and baseline methods are provided to understand their intrinsic difficulty. Multiple sensor-based robotic solutions are then evaluated, including hybrid force/motion control and 2D/3D pattern matching algorithms. An end-to-end integrated solution that accomplishes the tasks is also provided. The results and findings throughout the study reveal a few noticeable factors that impede the adoptions of the OTS solutions: expertise dependent, limited applicability, lack of interoperability, no scene awareness or error recovery mechanisms, and high cost. This paper also provides a first attempt of an objective benchmark performance on the NIST Assembly Task Boards as a reference comparison for future works on this problem.