LEGENT: Open Platform for Embodied Agents
Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu,, Pengkai Li, Lei Shi, Zhiyuan Liu, Maosong Sun

TL;DR
LEGENT is an open platform that facilitates the development of embodied agents by integrating large language and multimodal models within a scalable, interactive 3D environment, enabling advanced task performance and research collaboration.
Contribution
It introduces LEGENT, a comprehensive open platform combining a 3D environment, user interface, and data pipeline for embodied agent development using LLMs and LMMs.
Findings
Embodied agents trained on LEGENT data outperform GPT-4V in tasks.
The platform enables scalable data generation for embodied AI.
Promising generalization capabilities demonstrated in experiments.
Abstract
Despite advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), their integration into language-grounded, human-like embodied agents remains incomplete, hindering complex real-life task performance in physical environments. Existing integrations often feature limited open sourcing, challenging collective progress in this field. We introduce LEGENT, an open, scalable platform for developing embodied agents using LLMs and LMMs. LEGENT offers a dual approach: a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface, and a sophisticated data generation pipeline utilizing advanced algorithms to exploit supervision from simulated worlds at scale. In our experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks, showcasing promising generalization…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMulti-Agent Systems and Negotiation
