Event-Centric World Modeling with Memory-Augmented Retrieval for Embodied Decision-Making

TL;DR

This paper introduces an event-centric world modeling framework with memory-augmented retrieval for embodied decision-making, emphasizing interpretability, physical consistency, and real-time operation in dynamic environments.

Contribution

It proposes a novel structured environment representation and retrieval-based decision-making approach that enhances interpretability and physical consistency in autonomous agents.

Findings

Operates within real-time control constraints.

Maintains interpretable decision-making via case-based reasoning.

Ensures maneuvers are consistent with observed dynamics.

Abstract

Autonomous agents operating in dynamic and safety-critical environments require decision-making frameworks that are both computationally efficient and physically grounded. However, many existing approaches rely on end-to-end learning, which often lacks interpretability and explicit mechanisms for ensuring consistency with physical constraints. In this work, we propose an event-centric world modeling framework with memory-augmented retrieval for embodied decision-making. The framework represents the environment as a structured set of semantic events, which are encoded into a permutation-invariant latent representation. Decision-making is performed via retrieval over a knowledge bank of prior experiences, where each entry associates an event representation with a corresponding maneuver. The final action is computed as a weighted combination of retrieved solutions, providing a transparent link between decision and stored experiences. The proposed design enables structured abstraction of dynamic environments and supports interpretable decision-making through case-based reasoning. In addition, incorporating physics-informed knowledge into the retrieval process encourages the selection of maneuvers that are consistent with observed system dynamics. Experimental evaluation in UAV flight scenarios demonstrates that the framework operates within real-time control constraints while maintaining interpretable and consistent behavior.