CoEx -- Co-evolving World-model and Exploration

TL;DR

CoEx introduces a hierarchical, co-evolving world model and exploration strategy for LLM-based agents, enabling dynamic updates and improved planning in complex environments.

Contribution

It presents a novel hierarchical architecture with a neurosymbolic belief state that co-evolves with the world model, enhancing planning and exploration capabilities.

Findings

Outperforms existing agents in complex tasks

Effective dynamic updating of the world model

Improved planning and exploration results

Abstract

Planning in modern LLM agents relies on the utilization of LLM as an internal world model, acquired during pretraining. However, existing agent designs fail to effectively assimilate new observations into dynamic updates of the world model. This reliance on the LLM's static internal world model is progressively prone to misalignment with the underlying true state of the world, leading to the generation of divergent and erroneous plans. We introduce a hierarchical agent architecture, CoEx, in which hierarchical state abstraction allows LLM planning to co-evolve with a dynamically updated model of the world. CoEx plans and interacts with the world by using LLM reasoning to orchestrate dynamic plans consisting of subgoals, and its learning mechanism continuously incorporates these subgoal experiences into a persistent world model in the form of a neurosymbolic belief state, comprising textual inferences and code-based symbolic memory. We evaluate our agent across a diverse set of agent scenarios involving rich environments and complex tasks including ALFWorld, PDDL, and Jericho. Our experiments show that CoEx outperforms existing agent paradigms in planning and exploration.