Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation

TL;DR

This paper introduces an efficient video augmentation framework that converts simulated videos into realistic ones for vision-language-action models, improving real-world generalization.

Contribution

The authors propose a novel pipeline combining semantic extraction, caption rewriting, and a diffusion-based transfer model with acceleration techniques for scalable video augmentation.

Findings

Improves RDT-1B by 8% on Robotwin 2.0

Boosts $$ by 5.1% on LIBERO-Plus

Demonstrates consistent gains across multiple benchmarks

Abstract

Vision-language-action (VLA) models typically rely on large-scale real-world videos, whereas simulated data, despite being inexpensive and highly parallelizable to collect, often suffers from a substantial visual domain gap and limited environmental diversity, resulting in weak real-world generalization. We present an efficient video augmentation framework that converts simulated VLA videos into realistic training videos while preserving task semantics and action trajectories. Our pipeline extracts structured conditions from simulation via video semantic segmentation and video captioning, rewrites captions to diversify environments, and uses a conditional video transfer model to synthesize realistic videos. To make augmentation practical at scale, we introduce a diffusion feature-reuse mechanism that reuses video tokens across adjacent timesteps to accelerate generation, and a coreset sampling strategy that identifies a compact, non-redundant subset for augmentation under limited computation. Extensive experiments on Robotwin 2.0, LIBERO, LIBERO-Plus, and a real robotic platform demonstrate consistent improvements. For example, our method improves RDT-1B by 8% on Robotwin 2.0, and boosts $\pi_0$ by 5.1% on the more challenging LIBERO-Plus benchmark. Code is available at: https://github.com/nanfangxiansheng/Seeing-Realism-from-Simulation.