MotionBenchMaker: A Tool to Generate and Benchmark Motion Planning Datasets

TL;DR

MotionBenchMaker is an open-source tool that generates and benchmarks datasets for robot motion planning, facilitating fair comparison and accelerating research with a suite of prefabricated datasets.

Contribution

It introduces a flexible, easy-to-use platform for creating and benchmarking motion planning datasets, including a comprehensive set of 40 prefabricated datasets for common robots and environments.

Findings

Procedural dataset generation enables fair evaluation of planners.

The tool accelerates research by providing standardized datasets.

Benchmarking with MotionBenchMaker improves comparison of motion planning algorithms.

Abstract

Recently, there has been a wealth of development in motion planning for robotic manipulation new motion planners are continuously proposed, each with their own unique strengths and weaknesses. However, evaluating new planners is challenging and researchers often create their own ad-hoc problems for benchmarking, which is time-consuming, prone to bias, and does not directly compare against other state-of-the-art planners. We present MotionBenchMaker, an open-source tool to generate benchmarking datasets for realistic robot manipulation problems. MotionBenchMaker is designed to be an extensible, easy-to-use tool that allows users to both generate datasets and benchmark them by comparing motion planning algorithms. Empirically, we show the benefit of using MotionBenchMaker as a tool to procedurally generate datasets which helps in the fair evaluation of planners. We also present a suite of 40 prefabricated datasets, with 5 different commonly used robots in 8 environments, to serve as a common ground to accelerate motion planning research.