Creative Agents: Empowering Agents with Imagination for Creative Tasks

TL;DR

This paper introduces creative agents equipped with imagination capabilities, enabling them to generate diverse and novel solutions for open-ended tasks like building in Minecraft, surpassing previous instruction-following agents.

Contribution

It proposes a novel framework combining language and visual imagination with flexible controllers, and establishes a new benchmark for creative AI agents in open-world environments.

Findings

Creative agents achieve diverse building creation in Minecraft.

Imaginators improve the diversity and novelty of solutions.

New evaluation metrics effectively measure open-ended creativity.

Abstract

We study building embodied agents for open-ended creative tasks. While existing methods build instruction-following agents that can perform diverse open-ended tasks, none of them demonstrates creativity -- the ability to give novel and diverse solutions implicit in the language instructions. This limitation comes from their inability to convert abstract language instructions into concrete goals and perform long-horizon planning for such complicated goals. Given the observation that humans perform creative tasks with imagination, we propose a class of solutions, where the controller is enhanced with an imaginator generating detailed imaginations of task outcomes conditioned on language instructions. We introduce several approaches to implementing the components of creative agents. We implement the imaginator with either a large language model for textual imagination or a diffusion model for visual imagination. The controller can either be a behavior-cloning policy or a pre-trained foundation model generating executable codes in the environment. We benchmark creative tasks with the challenging open-world game Minecraft, where the agents create diverse buildings given free-form language instructions. We propose novel evaluation metrics for open-ended creative tasks utilizing GPT-4V, which holds many advantages over existing metrics. We perform a detailed experimental analysis of creative agents, showing that creative agents are the first AI agents accomplishing diverse building creation in the survival mode of Minecraft. Our benchmark and models are open-source for future research on creative agents (https://github.com/PKU-RL/Creative-Agents).