World2Act: Latent Action Post-Training via Skill-Compositional World Models

TL;DR

World2Act introduces a pixel-independent, contrastive learning framework for vision-language-action policies, leveraging skill decomposition and skill-compositional world models to enhance robustness and generalization in embodied agents.

Contribution

The paper presents a novel post-training approach that aligns actions with world model latents and introduces an LLM-based skill decomposition pipeline for flexible, temporally consistent skill execution.

Findings

Achieves state-of-the-art results on RoboCasa and LIBERO benchmarks.

Improves real-world embodied agent performance by 6.7%.

Enhances generalization and robustness of vision-language-action policies.

Abstract

World Models (WMs) have emerged as a promising approach for post-training Vision-Language-Action (VLA) policies to improve robustness and generalization under environmental changes. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to pixel-level artifacts and hallucination from imperfect WM rollouts. We introduce World2Act, a post-training framework that aligns VLA actions directly with WM video-dynamics latents using a contrastive matching objective, reducing dependence on pixels. Post-training performance is tied to rollout quality, yet current WMs struggle with arbitrary-length video generation as they are mostly trained on fixed-length clips while robotic execution durations vary widely. To address this, we propose an automatic LLM-based skill-decomposition pipeline that segments high-level instructions into low-level prompts. Our pipeline produces RoboCasa-Skill and LIBERO-Skill, supporting skill-compositional WMs that remain temporally consistent across diverse task horizons. Empirically, applying World2Act to VLAs like GR00T-N1.6 and Cosmos Policy achieves state-of-the-art results on RoboCasa and LIBERO, and improves real-world performance by 6.7%, enhancing embodied agent generalization.