AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation

TL;DR

AffordSim is a scalable simulation framework that generates affordance-aware data for robotic manipulation, improving task success and enabling zero-shot transfer to real robots.

Contribution

It introduces a novel data generator integrating open-vocabulary 3D affordance prediction for scalable, task-relevant robotic manipulation simulation.

Findings

Achieves 93% success rate compared to manual annotations on critical tasks.

Attains 89% success on complex composite tasks.

Zero-shot policies trained on AffordSim data reach 24% success on real robots.

Abstract

Many everyday robot manipulation skills are affordance-dependent, with success determined by whether the robot contacts the functional object region required by the subsequent action. Current simulation data generators obtain contacts from generic grasp estimators or per-object manual contact annotations, but generic estimators rank stable grasps without task semantics and often select contacts that are misaligned with the downstream action, while manual contact annotations must be rewritten for each new object and task. To solve these challenges, we introduce AffordSim, a scalable data generator and benchmark that integrates open-vocabulary 3D affordance prediction into simulation-based trajectory generation. Given a natural-language task description, AffordSim synthesizes a task-relevant scene, emits affordance queries, grounds them on object surfaces, samples region-conditioned grasps, and selects executable candidates with motion planning. It further randomizes object pose, texture, lighting, image noise, and cross-viewpoint backgrounds for sim-to-real transfer. We instantiate AffordSim as a 50-task benchmark across diverse manipulation skills, five robot embodiments, and 500+ rigid and articulated objects. AffordSim achieves 93% of the trajectory collection success rate of manual contact annotations on affordance-critical tasks and 89% on hard composite tasks. Vision-language-action policies trained on AffordSim data transfer zero-shot to a real Franka FR3, reaching 24% average success.