OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation

TL;DR

OCRTOC is a cloud-based benchmark and competition platform for robotic grasping and manipulation, enabling standardized, reproducible experiments on real robots and simulations to advance research in object rearrangement tasks.

Contribution

This work introduces a novel cloud-based benchmark and competition framework for robotic manipulation, providing standardized real robot setups and simulation tools to facilitate fair performance comparisons.

Findings

59 teams participated in the 2020 IROS competition

Standardized scenarios enable fair algorithm performance evaluation

The platform lowers barriers for reproducible robotic research

Abstract

In this paper, we propose a cloud-based benchmark for robotic grasping and manipulation, called the OCRTOC benchmark. The benchmark focuses on the object rearrangement problem, specifically table organization tasks. We provide a set of identical real robot setups and facilitate remote experiments of standardized table organization scenarios in varying difficulties. In this workflow, users upload their solutions to our remote server and their code is executed on the real robot setups and scored automatically. After each execution, the OCRTOC team resets the experimental setup manually. We also provide a simulation environment that researchers can use to develop and test their solutions. With the OCRTOC benchmark, we aim to lower the barrier of conducting reproducible research on robotic grasping and manipulation and accelerate progress in this field. Executing standardized scenarios on identical real robot setups allows us to quantify algorithm performances and achieve fair comparisons. Using this benchmark we held a competition in the 2020 International Conference on Intelligence Robots and Systems (IROS 2020). In total, 59 teams took part in this competition worldwide. We present the results and our observations of the 2020 competition, and discuss our adjustments and improvements for the upcoming OCRTOC 2021 competition. The homepage of the OCRTOC competition is www.ocrtoc.org, and the OCRTOC software package is available at https://github.com/OCRTOC/OCRTOC_software_package.