UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI

TL;DR

UnrealZoo offers a diverse set of photo-realistic virtual environments and tools to advance embodied AI research, highlighting the importance of environmental diversity and identifying current challenges in open-world scenarios.

Contribution

UnrealZoo provides a large collection of realistic virtual worlds and enhanced APIs, facilitating efficient data collection, environment augmentation, and benchmarking for embodied AI.

Findings

Environmental diversity improves RL generalization.

Embodied agents struggle with unstructured terrain and unseen morphologies.

Latency issues affect dynamic object interaction.

Abstract

We introduce UnrealZoo, a collection of over 100 photo-realistic 3D virtual worlds built on Unreal Engine, designed to reflect the complexity and variability of open-world environments. We also provide a rich variety of playable entities, including humans, animals, robots, and vehicles for embodied AI research. We extend UnrealCV with optimized APIs and tools for data collection, environment augmentation, distributed training, and benchmarking. These improvements achieve significant improvements in the efficiency of rendering and communication, enabling advanced applications such as multi-agent interactions. Our experimental evaluation across visual navigation and tracking tasks reveals two key insights: 1) environmental diversity provides substantial benefits for developing generalizable reinforcement learning (RL) agents, and 2) current embodied agents face persistent challenges in open-world scenarios, including navigation in unstructured terrain, adaptation to unseen morphologies, and managing latency in the close-loop control systems for interacting in highly dynamic objects. UnrealZoo thus serves as both a comprehensive testing ground and a pathway toward developing more capable embodied AI systems for real-world deployment.