ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulation

TL;DR

ManualVLA introduces a unified vision-language-action model with a planning and reasoning framework that improves robotic manipulation in complex, goal-oriented tasks by generating multimodal manuals and explicit control conditions.

Contribution

The paper presents ManualVLA, a novel Mixture-of-Transformers architecture that integrates manual generation and action execution, enabling better planning and manipulation in long-horizon tasks.

Findings

Achieves 32% higher success rate than previous SOTA on LEGO assembly.

Effectively generates multimodal manuals for complex tasks.

Utilizes a digital-twin toolkit for automatic manual data generation.

Abstract

Vision-Language-Action (VLA) models have recently emerged, demonstrating strong generalization in robotic scene understanding and manipulation. However, when confronted with long-horizon tasks that require defined goal states, such as LEGO assembly or object rearrangement, existing VLA models still face challenges in coordinating high-level planning with precise manipulation. Therefore, we aim to endow a VLA model with the capability to infer the "how" process from the "what" outcomes, transforming goal states into executable procedures. In this paper, we introduce ManualVLA, a unified VLA framework built upon a Mixture-of-Transformers (MoT) architecture, enabling coherent collaboration between multimodal manual generation and action execution. Unlike prior VLA models that directly map sensory inputs to actions, we first equip ManualVLA with a planning expert that generates intermediate manuals consisting of images, position prompts, and textual instructions. Building upon these multimodal manuals, we design a Manual Chain-of-Thought (ManualCoT) reasoning process that feeds them into the action expert, where each manual step provides explicit control conditions, while its latent representation offers implicit guidance for accurate manipulation. To alleviate the burden of data collection, we develop a high-fidelity digital-twin toolkit based on 3D Gaussian Splatting, which automatically generates manual data for planning expert training. ManualVLA demonstrates strong real-world performance, achieving an average success rate 32% higher than the previous hierarchical SOTA baseline on LEGO assembly and object rearrangement tasks.