Virtual Community: An Open World for Humans, Robots, and Society

TL;DR

This paper introduces Virtual Community, an open-world platform for studying human, robot, and society interactions using a physics-based simulator and real-world scenes, enabling new research challenges.

Contribution

The paper presents Virtual Community, a scalable, open-source multi-agent simulation platform grounded in real-world scenes for studying embodied social intelligence.

Findings

Evaluated baseline methods on community planning and robot collaboration challenges.

Demonstrated the platform's ability to simulate complex multi-agent interactions.

Highlighted the challenges in open-world multi-agent reasoning and cooperation.

Abstract

The rapid progress in AI and Robotics may lead to a profound societal transformation, as humans and robots begin to coexist within shared communities, introducing both opportunities and challenges. To explore this future, we present Virtual Community-an open-world platform for humans, robots, and society-built on a universal physics engine and grounded in real-world 3D scenes. With Virtual Community, we aim to enable the study of embodied social intelligence at scale. To support these, Virtual Community features: 1) An open-source multi-agent physics simulator that supports robots, humans, and their interactions within a society; 2) A large-scale, real-world aligned community generation pipeline, including vast outdoor space, diverse indoor scenes, and a community of grounded agents with rich characters and appearances. Leveraging Virtual Community, we propose two novel challenges. The Community Planning Challenge evaluates multi-agent reasoning and planning ability in open-world settings, such as cooperating to help agents with daily activities and efficiently connecting other agents. The Community Robot Challenge requires multiple heterogeneous robots to collaborate in solving complex open-world tasks. We evaluate various baselines on these tasks and demonstrate the challenges in both high-level open-world task planning and low-level cooperation controls. We hope that Virtual Community will unlock further study of human-robot coexistence within open-world environments.