VirtualEnv: A Platform for Embodied AI Research

TL;DR

VirtualEnv is an advanced simulation platform built on Unreal Engine 5 that enables detailed benchmarking of large language models in interactive, embodied scenarios involving object manipulation, navigation, and multi-agent collaboration.

Contribution

It introduces VirtualEnv, a versatile, open-source simulation environment supporting rich interactions and integration with LLMs and VLMs for embodied AI research.

Findings

Benchmarking of LLMs across complex tasks

Analysis of adaptability and multi-agent coordination

Procedural environment generation and validation

Abstract

As large language models (LLMs) continue to improve in reasoning and decision-making, there is a growing need for realistic and interactive environments where their abilities can be rigorously evaluated. We present VirtualEnv, a next-generation simulation platform built on Unreal Engine 5 that enables fine-grained benchmarking of LLMs in embodied and interactive scenarios. VirtualEnv supports rich agent-environment interactions, including object manipulation, navigation, and adaptive multi-agent collaboration, as well as game-inspired mechanics like escape rooms and procedurally generated environments. We provide a user-friendly API built on top of Unreal Engine, allowing researchers to deploy and control LLM-driven agents using natural language instructions. We integrate large-scale LLMs and vision-language models (VLMs), such as GPT-based models, to generate novel environments and structured tasks from multimodal inputs. Our experiments benchmark the performance of several popular LLMs across tasks of increasing complexity, analyzing differences in adaptability, planning, and multi-agent coordination. We also describe our methodology for procedural task generation, task validation, and real-time environment control. VirtualEnv is released as an open-source platform, we aim to advance research at the intersection of AI and gaming, enable standardized evaluation of LLMs in embodied AI settings, and pave the way for future developments in immersive simulations and interactive entertainment.