Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

TL;DR

This paper introduces DILLO, a fast, text-based world model that predicts action outcomes without visual simulation, enabling quicker and safer agent steering with significant speed improvements and success rate enhancements.

Contribution

The paper presents DILLO, a novel, language-based world model that replaces visual simulation with semantic descriptions, significantly accelerating proactive agent steering.

Findings

Achieves 14x speedup over visual simulation baselines.

Improves episode success rate by up to 15 percentage points.

Produces high-fidelity semantic descriptions of next states.

Abstract

Deploying safety-critical agents requires anticipating the consequences of actions before they are executed. While world models offer a paradigm for this proactive foresight, current approaches relying on visual simulation incur prohibitive latencies, often exceeding several seconds per step. In this work, we challenge the assumption that visual processing is necessary for failure prevention. We show that a trained policy's latent state, combined with its planned actions, already encodes sufficient information to anticipate action outcomes, making visual simulation redundant for failure prevention. To this end, we introduce DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from "simulate-then-act" to "describe-then-act." DILLO is trained via cross-modal distillation, where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes. This creates a text-only inference path, bypassing heavy visual generation entirely, achieving a 14x speedup over baselines. Experiments on MetaWorld and LIBERO demonstrate that DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 pp and 9.3 pp on average across tasks.