EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment

TL;DR

This paper introduces EmbodiedCity, a comprehensive benchmark platform with a realistic 3D simulation of a city environment to evaluate embodied AI agents in outdoor, real-world scenarios, addressing a gap in current indoor-focused research.

Contribution

It presents a high-fidelity city simulation platform with evaluation tasks and interfaces, enabling assessment of embodied AI in outdoor environments, expanding capabilities and practical applications.

Findings

Evaluated large language models' embodied intelligence capabilities.

Demonstrated the platform's potential for advancing embodied AI research.

Provided a versatile environment for future embodied AI development.

Abstract

Embodied artificial intelligence emphasizes the role of an agent's body in generating human-like behaviors. The recent efforts on EmbodiedAI pay a lot of attention to building up machine learning models to possess perceiving, planning, and acting abilities, thereby enabling real-time interaction with the world. However, most works focus on bounded indoor environments, such as navigation in a room or manipulating a device, with limited exploration of embodying the agents in open-world scenarios. That is, embodied intelligence in the open and outdoor environment is less explored, for which one potential reason is the lack of high-quality simulators, benchmarks, and datasets. To address it, in this paper, we construct a benchmark platform for embodied intelligence evaluation in real-world city environments. Specifically, we first construct a highly realistic 3D simulation environment based on the real buildings, roads, and other elements in a real city. In this environment, we combine historically collected data and simulation algorithms to conduct simulations of pedestrian and vehicle flows with high fidelity. Further, we designed a set of evaluation tasks covering different EmbodiedAI abilities. Moreover, we provide a complete set of input and output interfaces for access, enabling embodied agents to easily take task requirements and current environmental observations as input and then make decisions and obtain performance evaluations. On the one hand, it expands the capability of existing embodied intelligence to higher levels. On the other hand, it has a higher practical value in the real world and can support more potential applications for artificial general intelligence. Based on this platform, we evaluate some popular large language models for embodied intelligence capabilities of different dimensions and difficulties.