The ACRV Picking Benchmark (APB): A Robotic Shelf Picking Benchmark to Foster Reproducible Research

TL;DR

The paper introduces the ACRV Picking Benchmark (APB), a reproducible physical challenge for robotic shelf picking designed to standardize evaluation and foster progress in robotic manipulation research.

Contribution

It presents a new benchmark with detailed guidelines and an evaluation protocol for comprehensive robotic picking systems, including perception and manipulation.

Findings

Baseline system demonstrates feasibility on the benchmark.

Benchmark enables reproducible and comparable robotic picking research.

Provides detailed object set and arrangement guidelines.

Abstract

Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark (APB). Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of \emph{complete} robotic systems -- including perception and manipulation -- instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.