GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation

TL;DR

This paper introduces GenManip, a realistic simulation platform and benchmark for evaluating the generalization of instruction-following policies in robotic manipulation, emphasizing the role of foundation models and data scaling.

Contribution

It presents a new simulation platform and benchmark for studying policy generalization, along with an evaluation of modular versus end-to-end policies in diverse scenarios.

Findings

Modular systems with foundation models outperform end-to-end policies in generalization.

Data scaling benefits end-to-end policies more significantly.

GenManip enables systematic evaluation of policy adaptability in realistic settings.

Abstract

Robotic manipulation in real-world settings remains challenging, especially regarding robust generalization. Existing simulation platforms lack sufficient support for exploring how policies adapt to varied instructions and scenarios. Thus, they lag behind the growing interest in instruction-following foundation models like LLMs, whose adaptability is crucial yet remains underexplored in fair comparisons. To bridge this gap, we introduce GenManip, a realistic tabletop simulation platform tailored for policy generalization studies. It features an automatic pipeline via LLM-driven task-oriented scene graph to synthesize large-scale, diverse tasks using 10K annotated 3D object assets. To systematically assess generalization, we present GenManip-Bench, a benchmark of 200 scenarios refined via human-in-the-loop corrections. We evaluate two policy types: (1) modular manipulation systems integrating foundation models for perception, reasoning, and planning, and (2) end-to-end policies trained through scalable data collection. Results show that while data scaling benefits end-to-end methods, modular systems enhanced with foundation models generalize more effectively across diverse scenarios. We anticipate this platform to facilitate critical insights for advancing policy generalization in realistic conditions. Project Page: https://genmanip.axi404.top/.