Benchmarking Simulated Robotic Manipulation through a Real World Dataset

TL;DR

This paper introduces a benchmark for simulated robotic manipulation that uses a real-world dataset to evaluate and compare simulation environments, aiming to improve the realism and reliability of robotic simulations.

Contribution

It provides a novel benchmark framework based on a real-world dataset, including protocols and metrics, to assess the performance of simulation environments in robotic manipulation tasks.

Findings

PyBullet performs better on certain manipulation tasks.

V-Rep shows strengths in other manipulation scenarios.

The benchmark reveals specific deficiencies in current simulation environments.

Abstract

We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset. Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset. The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies. We apply our benchmark to two simulation environments, PyBullet and V-Rep, and publish the results. All materials required to benchmark an environment, including protocols and the dataset, can be found at the benchmarks' website https://research.csiro.au/robotics/manipulation-benchmark/.