SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds

TL;DR

SimWorld is a comprehensive, open-ended simulator built on Unreal Engine 5 that enables development and evaluation of large language model and vision-language model agents in realistic physical and social environments, supporting complex reasoning and interaction.

Contribution

The paper introduces SimWorld, a novel simulator with realistic physics, social dynamics, and multimodal interfaces designed specifically for training and testing advanced AI agents in complex, real-world-like scenarios.

Findings

LLM/VLM agents can perform long-horizon tasks in SimWorld.

Distinct reasoning patterns observed across different models.

SimWorld's environment and interface facilitate complex multi-agent interactions.

Abstract

While LLM/VLM-powered AI agents have advanced rapidly in math, coding, and computer use, their applications in complex physical and social environments remain challenging. Building agents that can survive and thrive in the real world (for example, by autonomously earning income or running a business) requires massive-scale interaction, reasoning, training, and evaluation across diverse embodied scenarios. However, existing world simulators for such development fall short: they often rely on limited hand-crafted environments, simulate simplified game-like physics and social rules, and lack native support for LLM/VLM agents. We introduce SimWorld, a new simulator built on Unreal Engine 5, designed for developing and evaluating LLM/VLM agents in rich, real-world-like settings. SimWorld offers three core capabilities: (1) realistic, open-ended world simulation, including accurate physical and social dynamics and language-driven procedural environment generation; (2) a rich interface for LLM/VLM agents, with multimodal world inputs and open-vocabulary actions at varying levels of abstraction; and (3) diverse and extensible physical and social reasoning scenarios that are easily customizable by users. We demonstrate SimWorld by deploying frontier LLM agents (e.g., GPT-4o, Gemini-2.5-Flash, Claude-3.5, and DeepSeek-Prover-V2) on long-horizon multi-agent delivery tasks involving strategic cooperation and competition. The results reveal distinct reasoning patterns and limitations across models. We open-source SimWorld and hope it becomes a foundational platform for advancing real-world agent intelligence across disciplines: https://simworld.org.