Towards Generalist Robots: A Promising Paradigm via Generative Simulation

TL;DR

This position paper proposes a generative simulation paradigm that leverages large-scale foundation models to create diverse training environments, aiming to develop generalist robots capable of performing various real-world tasks.

Contribution

It introduces the concept of generative simulation for robotics, using foundation models to generate training data and tasks at scale, facilitating the development of generalist robots.

Findings

Conceptual framework for generative simulation in robotics

Potential to scale up low-level skill learning

Discussion on resource challenges and collaborative opportunities

Abstract

This document serves as a position paper that outlines the authors' vision for a potential pathway towards generalist robots. The purpose of this document is to share the excitement of the authors with the community and highlight a promising research direction in robotics and AI. The authors believe the proposed paradigm is a feasible path towards accomplishing the long-standing goal of robotics research: deploying robots, or embodied AI agents more broadly, in various non-factory real-world settings to perform diverse tasks. This document presents a specific idea for mining knowledge in the latest large-scale foundation models for robotics research. Instead of directly using or adapting these models to produce low-level policies and actions, it advocates for a fully automated generative pipeline (termed as generative simulation), which uses these models to generate diversified tasks, scenes and training supervisions at scale, thereby scaling up low-level skill learning and ultimately leading to a foundation model for robotics that empowers generalist robots. The authors are actively pursuing this direction, but in the meantime, they recognize that the ambitious goal of building generalist robots with large-scale policy training demands significant resources such as computing power and hardware, and research groups in academia alone may face severe resource constraints in implementing the entire vision. Therefore, the authors believe sharing their thoughts at this early stage could foster discussions, attract interest towards the proposed pathway and related topics from industry groups, and potentially spur significant technical advancements in the field.