Real-Time Execution of Action Chunking Flow Policies

TL;DR

This paper introduces RTC, a novel inference-time algorithm that enables real-time, smooth, asynchronous execution of action chunking policies in AI systems, significantly reducing latency and improving performance in dynamic tasks.

Contribution

RTC is a new inference-time method that allows any diffusion- or flow-based VLA to run asynchronously without re-training, enhancing real-time performance and robustness.

Findings

RTC improves task throughput in dynamic environments.

RTC maintains high success rates despite inference delays.

RTC is effective in both simulated and real-world manipulation tasks.

Abstract

Modern AI systems, especially those interacting with the physical world, increasingly require real-time performance. However, the high latency of state-of-the-art generalist models, including recent vision-language action models (VLAs), poses a significant challenge. While action chunking has enabled temporal consistency in high-frequency control tasks, it does not fully address the latency problem, leading to pauses or out-of-distribution jerky movements at chunk boundaries. This paper presents a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies. Our method, real-time chunking (RTC), is applicable to any diffusion- or flow-based VLA out of the box with no re-training. It generates the next action chunk while executing the current one, "freezing" actions guaranteed to execute and "inpainting" the rest. To test RTC, we introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate 6 challenging real-world bimanual manipulation tasks. Results demonstrate that RTC is fast, performant, and uniquely robust to inference delay, significantly improving task throughput and enabling high success rates in precise tasks $\unicode{x2013}$ such as lighting a match $\unicode{x2013}$ even in the presence of significant latency. See https://pi.website/research/real_time_chunking for videos.