ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin, Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek, Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael, Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund,

TL;DR
ThreeDWorld (TDW) is an advanced platform for realistic, multi-modal physical simulation supporting interactive agents, diverse environments, and sensory data, facilitating research in vision, ML, and cognitive science.
Contribution
Introduction of TDW, a comprehensive platform enabling high-fidelity multi-modal simulation with customizable environments and agents for diverse research applications.
Findings
Enabled multi-modal scene understanding experiments
Supported physical dynamics prediction studies
Facilitated multi-agent interaction research
Abstract
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable agents that embody AI agents; and support for human interactions with VR devices. TDW's API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Multimodal Machine Learning Applications · Reinforcement Learning in Robotics
