Affordances Enable Partial World Modeling with LLMs

TL;DR

This paper demonstrates that large language models can serve as partial world models through affordances, enabling more efficient search and better task performance in robotics tasks by focusing on relevant actions and states.

Contribution

It formally proves large models can act as partial world models based on affordances and introduces distribution-robust affordances for multi-task settings.

Findings

Partial models improve search efficiency in robotics tasks.

Affordance-aware models outperform full models in reward achievement.

Reduced search branching factor with partial models.

Abstract

Full models of the world require complex knowledge of immense detail. While pre-trained large models have been hypothesized to contain similar knowledge due to extensive pre-training on vast amounts of internet scale data, using them directly in a search procedure is inefficient and inaccurate. Conversely, partial models focus on making high quality predictions for a subset of state and actions: those linked through affordances that achieve user intents~\citep{khetarpal2020can}. Can we posit large models as partial world models? We provide a formal answer to this question, proving that agents achieving task-agnostic, language-conditioned intents necessarily possess predictive partial-world models informed by affordances. In the multi-task setting, we introduce distribution-robust affordances and show that partial models can be extracted to significantly improve search efficiency. Empirical evaluations in tabletop robotics tasks demonstrate that our affordance-aware partial models reduce the search branching factor and achieve higher rewards compared to full world models.