MM-ACT: Learn from Multimodal Parallel Generation to Act

TL;DR

MM-ACT is a unified multimodal model that integrates vision, language, and action to enhance robotic task understanding and interaction, demonstrating high success rates across simulation and real-world tasks.

Contribution

The paper introduces MM-ACT, a novel unified VLA model with a shared token space and a new training paradigm for improved multimodal robotic learning.

Findings

Achieves 96.3% success on LIBERO simulation tasks.

Attains 72.0% success on real-robot Franka tasks.

Reaches 52.38% success on RoboTwin2.0 bimanual tasks.

Abstract

A generalist robotic policy needs both semantic understanding for task planning and the ability to interact with the environment through predictive capabilities. To tackle this, we present MM-ACT, a unified Vision-Language-Action (VLA) model that integrates text, image, and action in shared token space and performs generation across all three modalities. MM-ACT adopts a re-mask parallel decoding strategy for text and image generation, and employs a one-step parallel decoding strategy for action generation to improve efficiency. We introduce Context-Shared Multimodal Learning, a unified training paradigm that supervises generation in all three modalities from a shared context, enhancing action generation through cross-modal learning. Experiments were conducted on the LIBERO simulation and Franka real-robot setups as well as RoboTwin2.0 to assess in-domain and out-of-domain performances respectively. Our approach achieves a success rate of 96.3% on LIBERO, 72.0% across three tasks of real Franka, and 52.38% across eight bimanual tasks of RoboTwin2.0 with an additional gain of 9.25% from cross-modal learning. We release our codes, models and data at https://github.com/HHYHRHY/MM-ACT.