Action Agent: Agentic Video Generation Meets Flow-Constrained Diffusion

TL;DR

Action Agent introduces a two-stage framework combining language-guided video synthesis and flow-constrained diffusion to enhance multi-embodiment robot navigation success in simulation and real-world environments.

Contribution

It unifies agentic navigation video generation with flow-constrained diffusion control, achieving high success rates across multiple robot embodiments.

Findings

Video generation success increased from 35% to 86% across 50 tasks.

Achieved 73.2% success in simulation and 64.7% in real-world indoor navigation.

Operates at 40--47 Hz on a 43M-parameter model.

Abstract

We present Action Agent, a two-stage framework that unifies agentic navigation video generation with flow-constrained diffusion control for multi-embodiment robot navigation. In Stage I, a large language model (LLM) acts as an orchestration module that selects video diffusion models, refines prompts through iterative validation, and accumulates cross-task memory to synthesize physically plausible first-person navigation videos from language and image inputs. This increases video generation success from 35% (single-shot) to 86% across 50 navigation tasks. In Stage II, we introduce FlowDiT, a Flow-Constrained Diffusion Transformer that converts optimized goal videos and language instructions into continuous velocity commands using action-space denoising diffusion. FlowDiT integrates DINOv2 visual features, learned optical flow for ego-motion representation, and CLIP language embeddings for semantic stopping. We pretrain on the RECON outdoor navigation dataset and fine-tune on 203 Unitree G1 humanoid episodes collected in Isaac Sim to calibrate velocity dynamics. A single 43M-parameter checkpoint achieves 73.2% navigation success in simulation and 64.7% task completion on a real Unitree G1 in unseen indoor environments under open-loop execution, while operating at 40--47 Hz. We evaluate Action Agent across three embodiments: a Unitree G1 humanoid (real hardware), a drone, and a wheeled mobile robot (Isaac Sim), demonstrating that decoupling trajectory imagination from execution yields a scalable and embodiment-aware paradigm for language-guided navigation.