PoE-World: Compositional World Modeling with Products of Programmatic Experts

TL;DR

PoE-World introduces a novel method for modeling complex, non-gridworld environments using products of programmatic experts synthesized by LLMs, enabling efficient learning from sparse data and strong generalization in game environments.

Contribution

This work presents a new approach to world modeling by combining program synthesis with products of experts, extending application to complex, stochastic domains beyond natural language and grid worlds.

Findings

Effective modeling of complex, stochastic environments from few observations.

Demonstrated strong generalization to unseen game levels.

Achieved efficient performance in Atari game environments.

Abstract

Learning how the world works is central to building AI agents that can adapt to complex environments. Traditional world models based on deep learning demand vast amounts of training data, and do not flexibly update their knowledge from sparse observations. Recent advances in program synthesis using Large Language Models (LLMs) give an alternate approach which learns world models represented as source code, supporting strong generalization from little data. To date, application of program-structured world models remains limited to natural language and grid-world domains. We introduce a novel program synthesis method for effectively modeling complex, non-gridworld domains by representing a world model as an exponentially-weighted product of programmatic experts (PoE-World) synthesized by LLMs. We show that this approach can learn complex, stochastic world models from just a few observations. We evaluate the learned world models by embedding them in a model-based planning agent, demonstrating efficient performance and generalization to unseen levels on Atari's Pong and Montezuma's Revenge. We release our code and display the learned world models and videos of the agent's gameplay at https://topwasu.github.io/poe-world.