Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction

TL;DR

This paper introduces WMAct, a novel approach enabling large language models to internalize world models through efficient multi-turn interaction, leading to improved reasoning and task performance in complex environments.

Contribution

The paper proposes WMAct, a method that allows LLMs to internalize environmental dynamics via active reasoning and interaction strategies, reducing reliance on environmental cues.

Findings

WMAct enables single-turn task resolution in complex environments.

The approach improves transferability and reasoning benchmark performance.

It effectively internalizes world models, reducing interaction needs.

Abstract

Developing robust world model reasoning is crucial for large language model (LLM) agents to plan and interact in complex environments. While multi-turn interaction offers a superior understanding of environmental dynamics via authentic feedback, current approaches often impose a rigid reasoning process, which constrains the model's active learning, ultimately hindering efficient world model reasoning. To address these issues, we explore world-model internalization through efficient interaction and active reasoning (WMAct), which liberates the model from structured reasoning, allowing the model to shape thinking directly through its doing, and achieves effective and efficient world model reasoning with two key mechanisms: (1) a reward rescaling mechanism adjusting outcome reward based on action efficacy to incentivize redundancy reduction and purposeful interaction; (2) an interaction frequency annealing strategy to progressively reduce the maximum allowed interaction turns, which compels the model to condense its learning and internalize environmental dynamics rather than over-relying on environmental cues. Our experiments on Sokoban, Maze, and Taxi show that WMAct yields effective world model reasoning capable of resolving tasks in a single turn that previously required multiple interactions and fosters strong transferability to complex environments, improving performance on a suite of reasoning benchmarks.