Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification

TL;DR

Luban introduces autonomous embodied verification techniques to enhance open-ended creative AI agents, enabling them to perform complex creative tasks in Minecraft and real-world robotics by mimicking human verification processes.

Contribution

The paper presents a novel autonomous verification framework with visual and pragmatic checks, significantly improving creative task performance in AI agents.

Findings

Luban outperforms baselines by 33% to 100% in creative building tasks.

Extensive human studies validate the effectiveness of the verification approach.

Demonstrations show potential for physical-world robotic creation.

Abstract

Building open agents has always been the ultimate goal in AI research, and creative agents are the more enticing. Existing LLM agents excel at long-horizon tasks with well-defined goals (e.g., `mine diamonds' in Minecraft). However, they encounter difficulties on creative tasks with open goals and abstract criteria due to the inability to bridge the gap between them, thus lacking feedback for self-improvement in solving the task. In this work, we introduce autonomous embodied verification techniques for agents to fill the gap, laying the groundwork for creative tasks. Specifically, we propose the Luban agent target creative building tasks in Minecraft, which equips with two-level autonomous embodied verification inspired by human design practices: (1) visual verification of 3D structural speculates, which comes from agent synthesized CAD modeling programs; (2) pragmatic verification of the creation by generating and verifying environment-relevant functionality programs based on the abstract criteria. Extensive multi-dimensional human studies and Elo ratings show that the Luban completes diverse creative building tasks in our proposed benchmark and outperforms other baselines ($33\%$ to $100\%$) in both visualization and pragmatism. Additional demos on the real-world robotic arm show the creation potential of the Luban in the physical world.