RoboCAS: A Benchmark for Robotic Manipulation in Complex Object Arrangement Scenarios

TL;DR

RoboCAS introduces a new benchmark for robotic manipulation in complex object arrangement tasks, emphasizing long-horizon planning, spatial reasoning, and realistic physical simulation to evaluate and advance intelligent robotic agents.

Contribution

This paper presents RoboCAS, the first benchmark tailored for complex object arrangement scenarios, using scripted policies for diverse demonstrations and highlighting the limitations of current models.

Findings

Baseline models struggle with complex arrangements.

The benchmark reveals the need for improved long-horizon planning.

Realistic simulation enhances evaluation of robotic manipulation.

Abstract

Foundation models hold significant potential for enabling robots to perform long-horizon general manipulation tasks. However, the simplicity of tasks and the uniformity of environments in existing benchmarks restrict their effective deployment in complex scenarios. To address this limitation, this paper introduces the \textit{RoboCAS} benchmark, the first benchmark specifically designed for complex object arrangement scenarios in robotic manipulation. This benchmark employs flexible and concise scripted policies to efficiently collect a diverse array of demonstrations, showcasing scattered, orderly, and stacked object arrangements within a highly realistic physical simulation environment. It includes complex processes such as target retrieval, obstacle clearance, and robot manipulation, testing agents' abilities to perform long-horizon planning for spatial reasoning and predicting chain reactions under ambiguous instructions. Extensive experiments on multiple baseline models reveal their limitations in managing complex object arrangement scenarios, underscoring the urgent need for intelligent agents capable of performing long-horizon operations in practical deployments and providing valuable insights for future research directions. Project website: \url{https://github.com/notFoundThisPerson/RoboCAS-v0}.