SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes

TL;DR

This paper introduces SCENEREPLICA, a standardized, reproducible benchmark for real-world robot manipulation focusing on pick-and-place tasks using YCB objects, facilitating fair comparison of algorithms.

Contribution

It provides a new benchmark for robot manipulation that is easily reproducible and allows consistent evaluation of model-based and model-free grasping methods.

Findings

Benchmark enables fair comparison of manipulation algorithms.

Experimental results highlight strengths and weaknesses of different grasping approaches.

The framework promotes faster progress in robot manipulation research.

Abstract

We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on pick-and-place. Our benchmark uses the YCB objects, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. Additionally, the benchmark is designed to be easily reproducible in the real world, making it accessible to researchers and practitioners. We also provide our experimental results and analyzes for model-based and model-free 6D robotic grasping on the benchmark, where representative algorithms are evaluated for object perception, grasping planning, and motion planning. We believe that our benchmark will be a valuable tool for advancing the field of robot manipulation. By providing a standardized evaluation framework, researchers can more easily compare different techniques and algorithms, leading to faster progress in developing robot manipulation methods.