Enhancing Agent Learning through World Dynamics Modeling

TL;DR

This paper introduces DiVE, a framework that enables large language models to learn, verify, and evolve world dynamics from limited demonstrations, improving decision-making in interactive environments.

Contribution

The paper presents DiVE, a novel framework for discovering, verifying, and evolving world dynamics to enhance LLMs' understanding of environments.

Findings

LLMs guided by DiVE achieve human-level rewards in Crafter.

DiVE outperforms prior methods in MiniHack without task-specific training.

The framework effectively improves decision-making by modeling environment dynamics.

Abstract

Large language models (LLMs) have been increasingly applied to tasks in language understanding and interactive decision-making, with their impressive performance largely attributed to the extensive domain knowledge embedded within them. However, the depth and breadth of this knowledge can vary across domains. Many existing approaches assume that LLMs possess a comprehensive understanding of their environment, often overlooking potential gaps in their grasp of actual world dynamics. To address this, we introduce Discover, Verify, and Evolve (DiVE), a framework that discovers world dynamics from a small number of demonstrations, verifies the accuracy of these dynamics, and evolves new, advanced dynamics tailored to the current situation. Through extensive evaluations, we assess the impact of each component on performance and compare the dynamics generated by DiVE to human-annotated dynamics. Our results show that LLMs guided by DiVE make more informed decisions, achieving rewards comparable to human players in the Crafter environment and surpassing methods that require prior task-specific training in the MiniHack environment.