FetchBench: A Simulation Benchmark for Robot Fetching

TL;DR

FetchBench is a new simulation benchmark that evaluates robot fetching in complex environments, highlighting the challenges and current limitations of existing methods in diverse, realistic scenarios.

Contribution

The paper introduces FetchBench, a comprehensive benchmark with procedural scenes and a data pipeline, to evaluate and improve robot fetching methods beyond simple tabletop tasks.

Findings

Traditional methods achieve only 20% success rate on FetchBench

Identifies key bottlenecks in sense-plan-act pipeline

Provides recommendations for future improvements

Abstract

Fetching, which includes approaching, grasping, and retrieving, is a critical challenge for robot manipulation tasks. Existing methods primarily focus on table-top scenarios, which do not adequately capture the complexities of environments where both grasping and planning are essential. To address this gap, we propose a new benchmark FetchBench, featuring diverse procedural scenes that integrate both grasping and motion planning challenges. Additionally, FetchBench includes a data generation pipeline that collects successful fetch trajectories for use in imitation learning methods. We implement multiple baselines from the traditional sense-plan-act pipeline to end-to-end behavior models. Our empirical analysis reveals that these methods achieve a maximum success rate of only 20%, indicating substantial room for improvement. Additionally, we identify key bottlenecks within the sense-plan-act pipeline and make recommendations based on the systematic analysis.