FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making

TL;DR

FOUNDER integrates foundation models with world models to enable open-ended, reward-free embodied decision making, effectively grounding external observations in the agent's internal state for multi-task control.

Contribution

It introduces a novel framework that grounds foundation model representations in world models, enabling goal-conditioned policies learned through imagination in embodied environments.

Findings

Outperforms existing methods on multi-task visual control benchmarks.

Effectively captures deep semantics of tasks from text or videos.

Validates the consistency of the learned reward with ground-truth rewards.

Abstract

Foundation Models (FMs) and World Models (WMs) offer complementary strengths in task generalization at different levels. In this work, we propose FOUNDER, a framework that integrates the generalizable knowledge embedded in FMs with the dynamic modeling capabilities of WMs to enable open-ended task solving in embodied environments in a reward-free manner. We learn a mapping function that grounds FM representations in the WM state space, effectively inferring the agent's physical states in the world simulator from external observations. This mapping enables the learning of a goal-conditioned policy through imagination during behavior learning, with the mapped task serving as the goal state. Our method leverages the predicted temporal distance to the goal state as an informative reward signal. FOUNDER demonstrates superior performance on various multi-task offline visual control benchmarks, excelling in capturing the deep-level semantics of tasks specified by text or videos, particularly in scenarios involving complex observations or domain gaps where prior methods struggle. The consistency of our learned reward function with the ground-truth reward is also empirically validated. Our project website is https://sites.google.com/view/founder-rl.