AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learning

TL;DR

AnyTask introduces an automated framework combining GPU simulation and foundation models to generate diverse manipulation tasks and robot data, enabling effective sim-to-real policy learning with minimal human effort.

Contribution

The paper presents three novel agents for automated expert demonstration synthesis, advancing the efficiency of sim-to-real transfer in robot learning.

Findings

Policies achieve 44% success on real-world tasks

Automated data generation reduces human effort in task design

Framework generalizes to novel object poses

Abstract

Generalist robot learning remains constrained by data: large-scale, diverse, and high-quality interaction data are expensive to collect in the real world. While simulation has become a promising way for scaling up data collection, the related tasks, including simulation task design, task-aware scene generation, expert demonstration synthesis, and sim-to-real transfer, still demand substantial human effort. We present AnyTask, an automated framework that pairs massively parallel GPU simulation with foundation models to design diverse manipulation tasks and synthesize robot data. We introduce three AnyTask agents for generating expert demonstrations aiming to solve as many tasks as possible: 1) ViPR, a novel task and motion planning agent with VLM-in-the-loop Parallel Refinement; 2) ViPR-Eureka, a reinforcement learning agent with generated dense rewards and LLM-guided contact sampling; 3) ViPR-RL, a hybrid planning and learning approach that jointly produces high-quality demonstrations with only sparse rewards. We train behavior cloning policies on generated data, validate them in simulation, and deploy them directly on real robot hardware. The policies generalize to novel object poses, achieving 44% average success across a suite of real-world pick-and-place, drawer opening, contact-rich pushing, and long-horizon manipulation tasks. Our project website is at https://anytask.rai-inst.com .